Automate HTML form submission

I needed to make a large amount of changes that would require me to open a form, fill it in, submit it and repeat this hundreds of times. I really did not have the time to do this so I needed a quick way to automate this process.

First let's take a look at part of the form I needed to fill in:

<form method="POST" action="/cgi-bin/script.cgi">
<input type="radio" name="brand" value="product1">Product 1
<input type="radio" name="brand" value="product2">Product 2
<td>User: </td>
<td><input type="text" name="username" size="20" id="fp1"></td>
<input type="submit" value="Submit" name="submit">

So there's a radio button to select a product and a textbox to submit a username. The data is then posted to the script.cgi script. Now, how would we automate the process to fill this form from a script? I chose to use cURL to submit the data to the script like this:

curl -d "brand=pin&username=product1&username=user&submit=Submit" http://myserver.net/cgi-bin/script.cgi

The parameter -d tells cURL to HTTP post data. The data is specified between the quotes, first a value name and then the value data. Different values are seperated by the & character. Remember to figure out what part triggers submission of the code and to add this value as well. In my case "submit=Submit".

Now all you need to do is use your favorite method to call cURL and enter the correct data between the quotes. I used Bash and chose to read the data from CSV files.

One more thing: I needed to post a slash in one of the values ("server.net/user") and got strange results. As it turned out I needed to escape the data like this: "server.net%2Fuser". You can find a complete list of HTML escape characters here.