Curl, 17 years old today (opens in new tab)

(daniel.haxx.se)

420 pointsbagder11y ago95 comments

95 comments

72 comments · 16 top-level

wging11y ago· 18 in thread

I'm surprised it's so new. And wget is only a year older... what did people use before then?

Sometimes just plain telnet. It wasn't pretty, but it worked:

    $ telnet 80
    GET /

It should be noted that the above isn't a strict HTTP request header, it's missing quite a lot of detail, but it works as an example.

TazeTSchnitzel11y ago

Add two newlines and that is a perfectly valid HTTP/0.9 request.

josteink11y ago

Pre curl/wget that was probably sufficient.

HTTP/1.1 wasn't around until 1996 so back then Host: headers weren't even "optional". They simply didn't exist yet.

gsnedders11y ago

That's what HTTP/0.9 requests look like!

bagderOP11y ago

Those were the early days of HTTP, so no, people didn't really get a lot of HTTP "manually" like that. We used other existing tools to get gopher and ftp though.

When I started my work on httpget in 1996, I didn't know about wget and I didn't become aware of it until several years later....

hoboon11y ago

I would figure that by 1997 most people stopped using gopher/archive/veronica. I think I stopped by 1995 or so.

now days we use curl/wget for web dev it seems? probably used it for the same thing back then but I also image that lots of people just used their own ad-hoc scripts or whatever; i wish i had firsthand knowledge of that.

i used wget to download copies of websites when i found one that was mostly documents/images rather than webapps.

1 more reply

pjc5011y ago

In the HTTP/1.0 days I used telnet quite a bit. Or libwww-perl, when Perl was the language of choice for web applications. LWP has copyright notice:

Round about then I wrote something similar to WWW::Mechanize in order to automate access to the Orange Web-SMS gateway, which had an inconveniently deep login procedure.

tootie11y ago

Perl? I have only the vaguest recollection of those days. It looks like LWP is about 20 years old[1]. It's based on libwww which goes back to 1992[2].

[1] http://search.cpan.org/~ether/libwww-perl/ (see changelog)

[2] https://en.wikipedia.org/wiki/Libwww

zimpenfish11y ago

`lynx -dump` was a very useful tool.

72deluxe11y ago

I had to use that the other year whilst attempting to migrate an old old old Slackware box running on two Pentium 3s with an array of SCSI disks (so quite fancy for its time!). The machine had kernel sources that were different to the kernel running so no chance of compiling various bits of software, and I couldn't grab any GCC (EGCS?) binaries from anywhere.

I can't remember what I eventually did but I do remember discovering lynx -dump, so thanks for the reminder!

frabcus11y ago

I still can't stop myself typing `lynx -mime_header` to see server headers!

1 more reply

barsonme11y ago

How long have GET/POST commands been around? Probably not as long, but just curious.

castell11y ago

The first documented version of HTTP was HTTP V0.9 (1991) has only GET: http://www.w3.org/pub/WWW/Protocols/HTTP/AsImplemented.html

Basic HTTP as defined in 1992 had GET, PUT, HEAD, POST, LINK, TEXTSEARCH, CHECKIN, etc.: http://www.w3.org/Protocols/HTTP/Methods.html

More generic info: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

bagderOP11y ago

I believe they originally come from the libwww-perl project? According to wikipedia it was first released with perl 4.0036, which was in Februrary 1993, but I can't find out if it included GET and HEAD etc already then...

pdw11y ago

They're mentioned in a February 1996 changelog entry for libwww-perl, so at least 19 years.

purplezky11y ago

RFC 1945 dated may 1996 mentions GET and POST methods

http://www.rfc-base.org/txt/rfc-1945.txt

HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification reflects common usage of the protocol referred to as "HTTP/1.0".

userbinator11y ago

I used netcat - and still do for interacting with protocols in ways that curl doesn't support. AFAIK it was released in 1995, so it's only three years older than curl.

w8rbt11y ago

telnet and get

escherize11y ago· 11 in thread

I love curl so much. I just learned that you can 'copy to curl command' from the chrome inspector's network panel by right clicking on any request!!

I want to make a library that reads the curl command (and maybe request syntax?) and outputs a function that will do that command.

bagderOP11y ago

curl has a --libcurl option which you can add to a curl command line, and then it'll generate a libcurl-using C code template for that same operation...

72deluxe11y ago

That's incredibly useful, never knew that. Thanks!

jamessb11y ago

Similarly you can right-click and "Copy as cURL" from the Network pane of Firefox's web developer tools.

awalGarg11y ago

You can also edit requests and resend them from the FF dev tools. Alas, you can't (yet) do that from the Chrome dev tools.

artpar11y ago

https://shibukawa.github.io/curl_as_dsl/

TazeTSchnitzel11y ago

Of course for PHP, you can also just use the curl bindings ;)

DenisM11y ago

So many languages supported, but no plain C?

1 more reply

sorokod11y ago

That is very nice!

escherize11y ago

Wow! Thank you, what a great project!

anonfunction11y ago

We just built something[1] similar, but it uses the HAR request object to describe the request.

[1] https://github.com/Mashape/httpsnippet

organsnyder11y ago

Wow—that's incredibly useful. I didn't know about it either. Thanks for the tip!

nailer11y ago· 10 in thread

And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header. Curl just works.

leni53611y ago

> And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header.

Well I had to check, but this is not true at all:

   $ nc -l -p 9999
   GET / HTTP/1.1
   User-Agent: Wget/1.15 (linux-gnu)
   Accept: */*
   Host: localhost:9999
   Connection: Keep-Alive

Also wget can recursively mirror webpages and there are nice options to carefully select contents you want to download. It's quite dated though, I wish it could use an external downloader (like aria2) and only do the walking and converting links part itself.

sanqui11y ago

Do note that just a plain wget command isn't enough to properly archive/"back up" a webpage, because it doesn't save headers and other important metadata. Make sure to always use WARCs, which preserve metadata and have wonderful support, including the possibility to get absorbed into the Wayback Machine. More info:

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...

72deluxe11y ago

That mirroring option in wget has been very useful over the years, even more so if you remember to add --no-parent, else you'll end up with the ENTIRE site.

nailer11y ago

Sorry I stand corrected.

I stumbled upon this info when writing a redirector app recently. You're right, I should have verified that it was true.

ams611011y ago

wget to make a static copy of a site is the one thing I use it for. It's awesome. I tend to reach for curl first in most other cases, for one thing it's included in the "minimal" RHEL installation so I can pretty much depend on it always being there, whereas wget is extra.

baldfat11y ago

I love aria2c

I use it all the time even as the down loader for ArchLinux's pacman (package manager).

2 more replies

insertnickname11y ago

http://daniel.haxx.se/docs/curl-vs-wget.html

Piskvorrr11y ago

curl supports FTP, FTPS, Gopher, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMB/CIFS, SMTP, RTMP and RTSP. Wget only supports HTTP, HTTPS and FTP.

Ha! Good to know :)

josteink11y ago

Being this misinformed takes effort. I take it you haven't used wget the last decade?

nailer11y ago

It was a simple mistake, but the 1.1 support was added in 2011: http://en.m.wikipedia.org/wiki/Wget

kyberias11y ago· 9 in thread

> Rough estimates say we may have a billion users already.

This cannot be true by a long shot. Or am I missing something?

bagderOP11y ago

It all depends on how you count and what a "user" is of course, but this little list helps to give a picture: http://curl.haxx.se/docs/companies.html

anonfunction11y ago

The most common language used on websites is PHP which uses libcurl to handle HTTP requests.

kyberias11y ago

PHP applications may use libcurl when making outbound HTTP requests, right? That is only a small subset of these web sites. Most web sites just normally process HTTP requests.

This does not seem to explain the 1 billion users figure.

1 more reply

CJefferson11y ago

If you count a user as a system which has curl installed, and has run it at some point, is curl on iOS and Android? If so, you will be in that ball park I imagine.

Piskvorrr11y ago

I've seen it in various toasters, smart TVs and other embedded devices...so this is also a large "user" base.

hsx11y ago

That wouldn't be that surprising depending on what they mean. Assuming that Google or Facebook use curl in various applications, 1 billion sounds about right.

interurban11y ago

Depends if you're defining that as someone who has manually run a cURL command or as someone who has used an app/hardware that uses cURL "behind the scenes". If you use the latter I would say 1 billion might be a significant underestimate.

plorg11y ago

Isn't it installed on OS X by default?

kyberias11y ago

I don't know but just having it installed as part of OS distribution doesn't really count as "user", now does it?

xrstf11y ago· 3 in thread

> "If it doesn't load through curl, it's broken." --someone

So, so true. Thanks, curl.

arethuza11y ago

That's pretty much my own test for a Web based API - if I can drive it from the command line using curl then great, if I can't then it's broken.

laurent12345611y ago

Curl is great and all but I don't think this quote makes much sense. Are there examples of services that would, for example, load in a browser but not through curl? I'd think it's rather the opposite since you can through pretty much anything at curl and it will work.

cmg11y ago

Anything that requires JavaScript to fetch and display content.

2 more replies

pascalo11y ago· 2 in thread

Can I just say thank you for all those hours of hard work the maintainers have put in over the years?

ceejayoz11y ago

Yes, you can! Here's the link:

http://curl.haxx.se/donation.html

pascalo11y ago

Good call!

teamhappy11y ago· 1 in thread

    alias wget='echo "How dare you." && curl -O'
    brew rm wget

Happy birthday.

awalGarg11y ago

Also, we have aria2 already :P

josephpmay11y ago· 1 in thread

It's slightly weird to me that I am older than Curl and Wget. They always seemed like Unix Monoliths to me; I had just assumed they had always existed.

gobengo11y ago

THIS. I have the exact same feelings. Same with the 'Kubuntu is 10 years old' link on the homepage. I think we just grew up at the perfect time!

IshKebab11y ago· 1 in thread

Curl is pretty great to use, but be warned it has old smelly code and is probably full of security issues. I wouldn't use it for anything too critical.

charonn011y ago

That's sufficiently vague to be useless. Would you care to elaborate?

tlrobinson11y ago

Curl is great, but I also recently came across HTTPie (https://github.com/jakubroztocil/httpie) which has some nice features for playing around with HTTP APIs (JSON formatting, syntax highlighting, etc)

lloydde11y ago

I abuse curl most weeks. How many more web apps would fail at a header only request, if not for prodding from a curl user.

noselasd11y ago

Daniel Stenberg, the maintainer, was also on the packet pusher podcast recently, talking about HTTP/2 - http://packetpushers.net/show-224-http2-its-the-biggest-netw...

jkoudys11y ago

I was wgetting all my http requests until about 2 years ago. Curl's undeniable coolness won me over after 15 years. Now I practically live in curl when I'm setting up webservices, and libcurl for PHP does something on nearly every page request I have.

For PHP people, curl_multi_exec is the new event loop.

Pephers11y ago

curl is just awesome, thanks so much to the author and all maintainers over the years! It's still my go to application for testing and debugging HTTP requests.

cdnsteve11y ago

Happy bday!

awalGarg11y ago

cUrl, I am as old as you <3

j / k navigate · click thread line to collapse

95 comments

72 comments · 16 top-level

wging11y ago· 18 in thread

I'm surprised it's so new. And wget is only a year older... what did people use before then?

laumars11y ago

Sometimes just plain telnet. It wasn't pretty, but it worked:

    $ telnet 80
    GET /

It should be noted that the above isn't a strict HTTP request header, it's missing quite a lot of detail, but it works as an example.

TazeTSchnitzel11y ago

Add two newlines and that is a perfectly valid HTTP/0.9 request.

josteink11y ago

Pre curl/wget that was probably sufficient.

HTTP/1.1 wasn't around until 1996 so back then Host: headers weren't even "optional". They simply didn't exist yet.

gsnedders11y ago

That's what HTTP/0.9 requests look like!

bagderOP11y ago

Those were the early days of HTTP, so no, people didn't really get a lot of HTTP "manually" like that. We used other existing tools to get gopher and ftp though.

When I started my work on httpget in 1996, I didn't know about wget and I didn't become aware of it until several years later....

hoboon11y ago

I would figure that by 1997 most people stopped using gopher/archive/veronica. I think I stopped by 1995 or so.

i used wget to download copies of websites when i found one that was mostly documents/images rather than webapps.

1 more reply

pjc5011y ago

In the HTTP/1.0 days I used telnet quite a bit. Or libwww-perl, when Perl was the language of choice for web applications. LWP has copyright notice:

Round about then I wrote something similar to WWW::Mechanize in order to automate access to the Orange Web-SMS gateway, which had an inconveniently deep login procedure.

tootie11y ago

Perl? I have only the vaguest recollection of those days. It looks like LWP is about 20 years old[1]. It's based on libwww which goes back to 1992[2].

[1] http://search.cpan.org/~ether/libwww-perl/ (see changelog)

[2] https://en.wikipedia.org/wiki/Libwww

zimpenfish11y ago

`lynx -dump` was a very useful tool.

72deluxe11y ago

I can't remember what I eventually did but I do remember discovering lynx -dump, so thanks for the reminder!

frabcus11y ago

I still can't stop myself typing `lynx -mime_header` to see server headers!

1 more reply

barsonme11y ago

How long have GET/POST commands been around? Probably not as long, but just curious.

castell11y ago

The first documented version of HTTP was HTTP V0.9 (1991) has only GET: http://www.w3.org/pub/WWW/Protocols/HTTP/AsImplemented.html

Basic HTTP as defined in 1992 had GET, PUT, HEAD, POST, LINK, TEXTSEARCH, CHECKIN, etc.: http://www.w3.org/Protocols/HTTP/Methods.html

More generic info: http://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol

bagderOP11y ago

pdw11y ago

They're mentioned in a February 1996 changelog entry for libwww-perl, so at least 19 years.

purplezky11y ago

RFC 1945 dated may 1996 mentions GET and POST methods

http://www.rfc-base.org/txt/rfc-1945.txt

HTTP has been in use by the World-Wide Web global information initiative since 1990. This specification reflects common usage of the protocol referred to as "HTTP/1.0".

userbinator11y ago

I used netcat - and still do for interacting with protocols in ways that curl doesn't support. AFAIK it was released in 1995, so it's only three years older than curl.

w8rbt11y ago

telnet and get

escherize11y ago· 11 in thread

I love curl so much. I just learned that you can 'copy to curl command' from the chrome inspector's network panel by right clicking on any request!!

I want to make a library that reads the curl command (and maybe request syntax?) and outputs a function that will do that command.

bagderOP11y ago

curl has a --libcurl option which you can add to a curl command line, and then it'll generate a libcurl-using C code template for that same operation...

72deluxe11y ago

That's incredibly useful, never knew that. Thanks!

jamessb11y ago

Similarly you can right-click and "Copy as cURL" from the Network pane of Firefox's web developer tools.

awalGarg11y ago

You can also edit requests and resend them from the FF dev tools. Alas, you can't (yet) do that from the Chrome dev tools.

artpar11y ago

https://shibukawa.github.io/curl_as_dsl/

TazeTSchnitzel11y ago

Of course for PHP, you can also just use the curl bindings ;)

DenisM11y ago

So many languages supported, but no plain C?

1 more reply

sorokod11y ago

That is very nice!

escherize11y ago

Wow! Thank you, what a great project!

anonfunction11y ago

We just built something[1] similar, but it uses the HAR request object to describe the request.

[1] https://github.com/Mashape/httpsnippet

organsnyder11y ago

Wow—that's incredibly useful. I didn't know about it either. Thanks for the tip!

nailer11y ago· 10 in thread

And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header. Curl just works.

leni53611y ago

> And still better than wget, which only does http 1.0 and this has problems due to lacking a 'host' header.

Well I had to check, but this is not true at all:

   $ nc -l -p 9999
   GET / HTTP/1.1
   User-Agent: Wget/1.15 (linux-gnu)
   Accept: */*
   Host: localhost:9999
   Connection: Keep-Alive

sanqui11y ago

http://www.archiveteam.org/index.php?title=Wget_with_WARC_ou...

http://www.archiveteam.org/index.php?title=The_WARC_Ecosyste...

72deluxe11y ago

That mirroring option in wget has been very useful over the years, even more so if you remember to add --no-parent, else you'll end up with the ENTIRE site.

nailer11y ago

Sorry I stand corrected.

I stumbled upon this info when writing a redirector app recently. You're right, I should have verified that it was true.

ams611011y ago

baldfat11y ago

I love aria2c

I use it all the time even as the down loader for ArchLinux's pacman (package manager).

2 more replies

insertnickname11y ago

http://daniel.haxx.se/docs/curl-vs-wget.html

Piskvorrr11y ago

curl supports FTP, FTPS, Gopher, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS, FILE, POP3, IMAP, SMB/CIFS, SMTP, RTMP and RTSP. Wget only supports HTTP, HTTPS and FTP.

Ha! Good to know :)

josteink11y ago

Being this misinformed takes effort. I take it you haven't used wget the last decade?

nailer11y ago

It was a simple mistake, but the 1.1 support was added in 2011: http://en.m.wikipedia.org/wiki/Wget

kyberias11y ago· 9 in thread

> Rough estimates say we may have a billion users already.

This cannot be true by a long shot. Or am I missing something?

bagderOP11y ago

It all depends on how you count and what a "user" is of course, but this little list helps to give a picture: http://curl.haxx.se/docs/companies.html

anonfunction11y ago

The most common language used on websites is PHP which uses libcurl to handle HTTP requests.

kyberias11y ago

PHP applications may use libcurl when making outbound HTTP requests, right? That is only a small subset of these web sites. Most web sites just normally process HTTP requests.

This does not seem to explain the 1 billion users figure.

1 more reply

CJefferson11y ago

If you count a user as a system which has curl installed, and has run it at some point, is curl on iOS and Android? If so, you will be in that ball park I imagine.

Piskvorrr11y ago

I've seen it in various toasters, smart TVs and other embedded devices...so this is also a large "user" base.

hsx11y ago

That wouldn't be that surprising depending on what they mean. Assuming that Google or Facebook use curl in various applications, 1 billion sounds about right.

interurban11y ago

plorg11y ago

Isn't it installed on OS X by default?

kyberias11y ago

I don't know but just having it installed as part of OS distribution doesn't really count as "user", now does it?

xrstf11y ago· 3 in thread

> "If it doesn't load through curl, it's broken." --someone

So, so true. Thanks, curl.

arethuza11y ago

That's pretty much my own test for a Web based API - if I can drive it from the command line using curl then great, if I can't then it's broken.

laurent12345611y ago

cmg11y ago

Anything that requires JavaScript to fetch and display content.

2 more replies

pascalo11y ago· 2 in thread

Can I just say thank you for all those hours of hard work the maintainers have put in over the years?

ceejayoz11y ago

Yes, you can! Here's the link:

http://curl.haxx.se/donation.html

pascalo11y ago

Good call!

teamhappy11y ago· 1 in thread

    alias wget='echo "How dare you." && curl -O'
    brew rm wget

Happy birthday.

awalGarg11y ago

Also, we have aria2 already :P

josephpmay11y ago· 1 in thread

It's slightly weird to me that I am older than Curl and Wget. They always seemed like Unix Monoliths to me; I had just assumed they had always existed.

gobengo11y ago

THIS. I have the exact same feelings. Same with the 'Kubuntu is 10 years old' link on the homepage. I think we just grew up at the perfect time!

IshKebab11y ago· 1 in thread

Curl is pretty great to use, but be warned it has old smelly code and is probably full of security issues. I wouldn't use it for anything too critical.

charonn011y ago

That's sufficiently vague to be useless. Would you care to elaborate?

tlrobinson11y ago

lloydde11y ago

I abuse curl most weeks. How many more web apps would fail at a header only request, if not for prodding from a curl user.

noselasd11y ago

Daniel Stenberg, the maintainer, was also on the packet pusher podcast recently, talking about HTTP/2 - http://packetpushers.net/show-224-http2-its-the-biggest-netw...

jkoudys11y ago

For PHP people, curl_multi_exec is the new event loop.

Pephers11y ago

curl is just awesome, thanks so much to the author and all maintainers over the years! It's still my go to application for testing and debugging HTTP requests.

cdnsteve11y ago

Happy bday!

awalGarg11y ago

cUrl, I am as old as you <3

j / k navigate · click thread line to collapse