Talking to the Web: curl

curl fetches whatever is at a web address, from the terminal or a script: it prints a page, saves a download with -o or -O, asks a web API for JSON, shows the status code with -w, and with -f fails on an error instead of saving the error page.

Journeyman · Unix · Linux · Updated

After this lesson you can

  • Get a web page or an API's JSON with curl, and save it to a file with -o or -O
  • Check a request's status code with -w, and make curl fail on an HTTP error with -f

Key takeaways

  • curl ADDRESS prints what's at a web address. -o FILE saves it under a name you choose, and -O under the name at the end of the address.
  • -s hides the progress meter. Put an address in single quotes, so a ? or & in it stays part of the address.
  • An API answers with data, often JSON. weather.gov asks for a User-Agent that says who's asking: -A 'my-app (you@example.com)'.
  • Every answer has a status code: 200 worked, 404 no such page. -w '%{http_code} ' prints it.
  • Without -f, curl saves an error page and reports success. curl -fsS fails with exit code 22 and a message, and saves nothing.

A tutorial says "download the installer with curl". A server you look after has no browser at all. And the forecast saved in JSON with jq is already out of date: a script that reports the weather every morning has to fetch a fresh one. curl (read "curl", from client for URLs) gets whatever is at a web address, from the terminal or from a script. Web developers use it all day to try the APIs they build, and administrators use it to download tools and to check that a site is up. It comes with Linux, macOS and Ubuntu in WSL, so there's nothing to install.

The examples run in the weather folder from JSON with jq (cd ~/terminal-practice/weather), and they need the internet.

A page, and a download

Give curl a web address, a URL, and it prints what's there:

curl https://example.com

<!doctype html><html lang=en><head><meta charset=utf-8><link rel=icon href=data:,><meta name=viewport content="width=device-width,initial-scale=1"><title>Example Domain</title><style>html{color-scheme:light dark;background:light-dark(#eee,#222)}body{font:16px/1.6 system-ui,sans-serif;max-width:26em;margin:auto;padding:25vh 2em 2em;text-align:center}</style></head><body><p>This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.</p><script src=/s.js></script></body></html>

That's the page's HTML, the text a browser turns into the page you see. example.com is a site kept for examples like this one. To keep what you fetch, -o (read "output") names the file to save it in, and -O (a capital O) uses the name at the end of the address. While it saves, curl draws a progress meter; -s (silent) hides it:

curl -s -o example.html https://example.com
curl -sO https://example.com/index.html
ls

example.html  forecast.json  index.html

That's how install instructions download a tool, a script or a zip file.

An API answers with data

An API is a web address made for programs: it answers with data instead of a page. The US National Weather Service runs a free one at api.weather.gov, with no account and no key. A forecast takes two steps. First ask which forecast covers a spot on the map, given as latitude and longitude (here, Central Park). The answer is JSON, and jq from the last lesson picks out the forecast's own address:

curl -s -A 'weather-lesson (sam@example.com)' 'https://api.weather.gov/points/40.7829,-73.9654' | jq -r '.properties.forecast'

https://api.weather.gov/gridpoints/OKX/34,45/forecast

-A sets the User-Agent, a line every request carries that says which program is asking. weather.gov asks every program for one with a way to reach its owner, so put your own app's name and email address there. The single quotes keep the address in one piece: a ?, & or * in an address means something to the shell. Then ask the forecast's address and save the answer:

curl -s -A 'weather-lesson (sam@example.com)' -o forecast-live.json 'https://api.weather.gov/gridpoints/OKX/34,45/forecast'
jq -r '.properties.periods[] | "\(.name): \(.shortForecast)"' forecast-live.json | head -n 3

Today: Mostly Sunny
Tonight: Mostly Clear
Tuesday: Mostly Sunny

Yours will be today's forecast. An address can carry options after a ? (read "here come the options"), each a name, = and a value, joined by & (read "and"): the query string. ?units=si asks for the metric system:

curl -s -A 'weather-lesson (sam@example.com)' 'https://api.weather.gov/gridpoints/OKX/34,45/forecast?units=si' | jq -r '.properties.periods[0] | "\(.name): \(.temperature) \(.temperatureUnit)"'

Today: 14 C

weather.gov covers only the United States; outside it, practice on the saved forecast.json. And ask once, then work with the saved answer: an API is someone else's computer.

Did it work? Status codes

Every answer from a web server comes with a three-digit status code: 200 means it worked, 404 means there's no such page, and 500 means the server itself broke. -w (read "write out") prints facts about the request when it's done. '%{http_code}\n' is the status code and a new line, and -o /dev/null throws the page away (Three Streams: Output, Errors and Input):

curl -s -o /dev/null -w '%{http_code}\n' https://example.com
curl -s -o /dev/null -w '%{http_code}\n' https://example.com/no-such-page

200
404

A web developer checks a site this way after every change, and a script can check for 200 before it trusts an answer.

Stop on errors: -f

Now the trap. Ask for a page that isn't there, the way a script with a typo in its address would:

curl -s -o new-forecast.json https://example.com/no-such-page
echo "exit code: $?"

exit code: 0

Exit code 0 means success (Did It Work? Exit Codes, && and ||), and new-forecast.json now holds a web page instead of a forecast. To curl, the server answered, so it worked. -f (fail) changes that: when the status code is an error, curl stops with exit code 22 and saves nothing. With -s, add -S (a capital S) so the error message still shows:

curl -fsS -o new-forecast.json https://example.com/no-such-page
echo "exit code: $?"

curl: (22) The requested URL returned error: 404
exit code: 22

In a script, || (read "or else") decides what happens when the fetch fails. Names ending in .invalid never exist, so this one fails every time, the way it would with no internet:

curl -fsS -A 'weather-lesson (sam@example.com)' -o forecast.json 'https://forecast.invalid/today' || echo "No new forecast; keeping the saved one"
jq -r '.properties.periods[0].name' forecast.json

curl: (6) Could not resolve host: forecast.invalid
No new forecast; keeping the saved one
Today

The saved forecast is still there, untouched. Write curl -fsS in every script.

Check yourself

2 questions to try on the lesson page.

So what

With curl, any web address is one line away from your terminal: a page, a download, or an API's data for a script, and -f makes sure a failure looks like one.

Key terms

Cheat sheets

Related