You can use the cURL library and the
curl command to design your own Request and explore the Response. There are many possible uses like e.g., API debug, web hacking, pen testing.
curl is a tool to transfer data from or to a server, using one of the supported protocols (e.g., FTP, GOPHER, HTTP, HTTPS, IMAP, LDAP, POP3, RTMP, SCP, SFTP, SMTP, TELNET). The command is designed to work without user interaction.
curl offers a busload of useful tricks like proxy support, user authentication, FTP upload, HTTP post, SSL connections, cookies, file transfer resume, Metalink, and more. As you will see below, the number of features will make your head spin!
curl is a truly powerful command, however it does at the cost of complexity. Here I will show some real-world use cases.
The URL syntax is protocol-dependent. If you specify URL without
curl will attempt to guess what protocol you might want. It will then default to HTTP but try other protocols based on often-used host name prefixes. For example, for host names starting with “ftp.” curl will assume you want to speak FTP.
You can specify multiple URLs or parts of URLs by writing part sets within braces as in:
or you can get sequences of alphanumeric series by using
[ ] as in:
Nested sequences are not supported, but you can use several ones next to each other:
You can specify any amount of URLs on the command line. They will be fetched in a sequential manner in the specified order.
You can specify a step counter for the ranges to get every Nth number or letter:
In order to analyze in depth what we send and receive we might save everything on a file, this is as easy as:
Save To Disk
If you want save the Response to disk you can use option
-o <file>. If you are using
 to fetch multiple documents, you can use ‘
#‘ followed by a number in the specifier. That variable will be replaced with the current string for the URL being fetched. Remember to protect the URL from shell by adding quotes if you receive the error message
internal error: invalid pattern type (0). Examples:
-O writes output to a local file named like the remote file we get (only the file part of the remote file is used, the path is cut off). The remote file name to use for saving is extracted from the given URL, nothing else. Consequentially, the file will be saved in the current working directory. If you want the file saved in a different directory, make sure you change current working directory before you invoke curl:
Only the file part of the remote file is used, the path is cut off, thus the file will be saved as
Set HTTP Request Method
curl default HTTP method, GET, can be set to any method you would like using the
-X <command> option. The usual suspects POST, PUT, DELETE, and even custom methods, can be specified:
Normally you don’t need this option. All sorts of GET, HEAD, POST and PUT requests are rather invoked by using dedicated command line options.
Forms are the general way a web site can present a HTML page with fields for
the user to enter data in, and then press some kind of ‘submit’
button to get that data sent to the server. The server then typically uses
the posted data to decide how to act. Like using the entered words to search
in a database, or to add the info in a bug track system, display the entered
address on a map or using the info as a login-prompt verifying that the user
is allowed to see what it is about to see.
-d option we can specify URL encoded field names and values:
A very common way for HTML based application to pass state information between pages is to add hidden fields to the forms. Hidden fields are already filled in, they aren’t displayed to the user and they get passed along just as all the other fields. To
curl there is no difference at all, you just need to add it on the command line.
Set Request Headers
Request headers allow clients to provide servers with meta information about things such as authorization, capabilities, and body content-type. OAuth2 uses an
Authorization header to pass access tokens, for example. Custom headers are set in curl using the
Note that if you should add a custom header that has the same name as one of the internal ones curl would use, your externally set header will be used instead of the internal one. You should not replace internally set headers without knowing perfectly well what you’re doing. Remove an internal header by giving a replacement without content on the right side of the colon, as in:
If you send the custom header with no-value then its header must be terminated with a semicolon, such as
-H "X-Custom-Header;" to send
curl will make sure that each header you add/replace is sent with the proper end-of-line marker, you should thus not add that as a part of the header content: do not add newlines or carriage returns, they will only mess things up for you.
A HTTP request may include a
referer field (yes it is misspelled), which can be used to tell from which URL the client got to this particular resource. Some programs/scripts check the referer field of requests to verify that this wasn’t arriving from an external site or an unknown page. While this is a stupid way to check something so easily forged, many scripts still do it.
This can also be set with the
-H, --header flag of course. When used with
-L, --location you can append
";auto" to the
--referer URL to make curl automatically set the previous URL when it follows a
Location: header. The
";auto" string can be used alone, even if you don’t set an initial
To specify the User-Agent string to send to the HTTP server you can use
--user-agent flag. To encode blanks in the string, surround the string with single quote marks. This can also be set with the
-H, --header option of course. Many applications use this information to decide how to display pages. At times, you will see that getting a page with curl will not return the same page that you see when getting the page with your browser. Then you know it is time to set the User Agent field to fool the server into thinking you’re one of those browsers:
The way the web browsers do “client side state control” is by using cookies. Cookies are just names with associated contents. The cookies are sent to the client by the server. The server tells the client for what path and host name it wants the cookie sent back, and it also sends an expiration date and a few more properties.
When a client communicates with a server with a name and path as previously specified in a received cookie, the client sends back the cookies and their contents to the server, unless of course they are expired.
Many applications and servers use this method to connect a series of requests into a single logical session. To be able to use curl in such occasions, we must be able to record and send back cookies the way the web application expects them. The same way browsers deal with them.
It is supposedly the data previously received from the server in a
"Set-Cookie:" line. The data should be in the format
= symbol is used in the line, it is treated as a filename to use to read previously stored cookie lines from, which should be used in this session if they match. Using this method also activates the “cookie parser” which will make curl record incoming cookies too, which may be handy if you’re using this in combination with the
-L, --location option. The file format of the file to read cookies from should be plain HTTP headers or the Netscape/Mozilla cookie file format. NOTE that the file specified with
-b, --cookie is only used as input. No cookies will be stored in the file. To store cookies, use the
-c, --cookie-jar option or you could even save the HTTP headers to a file using
Work In Progress…
Ok there are many more options, but I will stop here for now. I will add something in the future, so if you have any request (like using more real urls) just leave a comment.
This is a legendary tool developed by the author of The Web Application Hacker’s Handbook: Finding and Exploiting Security Flaws (2nd edition).
Unfortunately there is
no native Mac Os X version but Corsaire packed one. Since they updated the site, the link to download it provided in the aforementioned book is broken, so I will provide a new working one of the recently released version 1.5 1.6 of Burp Suite Free.
All rights reserved to Corsair and Portswigger.
See also here for a [much less powerful] alternative.
Paros is a web proxy that allows to intercept and modify all HTTP and HTTPS data between server and client, including cookies and form fields.
I will not explain how and why you use it, but if you are really interested, please take a look on the book: The Web Application Hacker’s Handbook: Discovering and Exploiting Security Flaws.
Unfortunately there is no native Mac Os X version but Corsaire packed one. Since they updated the site, the link to download it provided in the aforementioned book is broken, so I will provide a new working one. All rights reserved to Corsair and Paros Team.
Paros is no longer developed (it also requires JDK 6 which is not longer supported).
An alternative is a fork of it, Zed Attack Proxy, maintained by OWASP:
Zed Attack Proxy (ZAP)
You are accustomed to incomprehensible command line tools???
Finally I have the right solution: IP Scanner Pro
It’s all about friendlyness!!! You can ping, wake up, insert into whitelist, etc all the devices found with just one click.
I will show you just an image, you don’t need anything else! 😉
NOTE: I have hidden MAC address.
MAC Address is a unique identifier of 48 bits assigned to network interfaces.
This Address is unique in the world, so it does identify you, everywhere.
From a Security & Privacy perspective this is not good at all!!!
How to modify it:
- Open Terminal app.
sudo ifconfig [en0-en1] ether [MAC Address]
- where you can choose en0 for wired ethernet interface or en1 for airport interface
- MAC Address is something like FF:FF:FF:FF:FF:FF
where there are 6 groups of 2 exadecimal symbols separated by colons (:)
- Insert your Administrator password.
- Go to Network System Preferences, select active interface and push Advanced… button.
- In TCP/IP tab click on Renew DHCP Lease button.
- You should now see your new MAC Address in Hardware tab.
NOTE: you can set Configure: Manually in Hardware tab to manually set MAC Address and avoid using Terminal.
NOTE2: if you do not renew DHCP Lease you may loose network connectivity.
NOTE3: if you have set MAC Address through Terminal it will be restored to it own default after reboot or shutdown.
NOTE4: some Hotels or private Acces Point use MAC Filtering to prevent abuse of their network. The changing of MAC Address allow you circumviate this kind of restrictions (see MAC Spoofing). This is NOT legitimate, so if you are not an evil user and you need emergency network access, try to set your MAC Address to FF:FF:FF:FF:FF:FF (could not work of course). This way System Administrator will readily understand what is going on. 😉