My firsts tests with Chess.com API (some examples with JSON manipulations)

Hi. Thanks for releasing this API, this is really cool and I am looking forward getting more API calls available (e.g. PGN, and capability of course to send moves).

Today, I have played a little bit with the API to get JSON results with curl and parse it with JQ parser (https://stedolan.github.io/jq/) and I thought it could be good to share my few lines of codes using with you. I may inspire others...

NB: please change the environment variable $CHESSCOM_USERNAME to your own username in the code below 

Example #1: get my games of the current month

$ CHESSCOM_USERNAME="pmaugeri"

$ curl -k https://api.chess.com/pub/player/$CHESSCOM_USERNAME/games/`date +%Y`/`date +%m`

Example #2 (with JQ): get only the "3day" time-class games of the month

$ curl -k https://api.chess.com/pub/player/$CHESSCOM_USERNAME/games/`date +%Y`/`date +%m` | jq '.games[] | select(.time_class=="3day")'

Example #3: return only the fen field

$ curl -k https://api.chess.com/pub/player/$CHESSCOM_USERNAME/games/`date +%Y`/`date +%m` | jq '.games[] | select(.time_class=="3day") | .fen'

Example #4: return only the games I won

$ curl -k https://api.chess.com/pub/player/$CHESSCOM_USERNAME/games/`date +%Y`/`date +"%m"` | jq --arg USERNAME $CHESSCOM_USERNAME '.games[] | select(.time_class=="3day") | (select(.white["result"] == "win" and .white["@id"] == "https://api.chess.com/pub/player/"+$USERNAME )), (select(.black["result"] == "win" and .black["@id"] == "https://api.chess.com/pub/player/"+$USERNAME )) | .fen'

That's great work! I just recently found out about jq and it looks super useful. Your examples are really helping picture the data we're delivering. You could post these to https://jqplay.org/ and then other readers could just click over and try them out.

Once the endpoints are finished up and stabilized, we plan to build a few tools to help people dive in. For the read-only API, you've shown that these don't need to be complex SDKs. Aside from `jq`, are there other tools you use to browse APIs? We were thinking of sharing Postman collections and perhaps other generic clients.

Yes, postman is a tool that I use very often too when working with APIs. 

@pmaugeri we have created a JSON collection of all our current endpoints that you can find in this section of the documentation. 

First time I've needed to use https and not http. My preferred scripting language is perl (I've used others, but I started with perl in 1989 ) and I've used LWP before.

Trying to get:

https://api.chess.com/pub/player/skelos/games

gives me  403 refused error, and debugging shows that Cloudflare have stopped me.

<title>Access denied | api.chess.com used Cloudflare to restrict access</title>

I can/will dig into this more tomorrow, and in the meantime can download using tnftp (previously lukemftp) without problems, and also use Chrome without problems (but then I'm logged in, so it's not a fair test).

Please, nobody spend any time on this: if you recognise the problem it'll save me some time; otherwise I go digging and figure it out myself.

With the downloaded JSON I can parse it just fine and get the data I want, but it'll be simpler when I have the script that wants the data fetch it.

Best regards to all,

Giles

Details for anyone interested in the perl aspects:

1. I'm on OS X, but built the latest release perl version 5.26.1

2. I've built the current release of OpenSSL and have LWP using it

3. For simplicity (in my usages, parsing the data is not going to be a bottleneck, although you know what they say about programmers and performance predictions) I'm using JSON:: PP (no space there, but cut out the emoticons!) which ships with perl. One less thing to install.

I cut my teeth on Perl, and have a rather worn-out O'Reilly camel. I still use regular expressions as my go-to can opener.

HTTPS will be enforced on all our domains once we can get a few stragglers updated, maybe in a month. This will allow us access to certain HTTP/2 techniques that should speed a lot of things up. Since these endpoints publish public data for unauthenticated access, it seems quite unnecessary for security reasons — the only benefit is that you know the data are coming from us. Do you think that regular HTTP might be a requirement for certain scripting?

Cloudflare should not have interfered at all. Was it just that URL, or all of them? If you could provide full request/response headers that would help us a lot. I'd like to make sure they get out of the way of anything you are trying to do — very likely they have "DDoS security features" that block scripted access, but that's exactly what we are trying to encourage! I'd like to fine-tune that, if you have additional info to share.

OK. Today is "maybe", but I'll definitely get onto it.

I would like to use perl, HTTPS is no problem at all, and as I hadn't set the user agent string it's possible they said "Ah, script, no!" Which eventually might be good for www.chess.com after encouraging the web-scrapers to migrate to the public APIs (at least the read-only ones), but is not good as you note for api.chess.com.

Sightly interesting if they're blocking perl's LWP but not curl, wget and tnftp/lukemftp.

Headers will explain, so later I'll get headers. There should be debugging options to show them to me; I'll be peeved if I have to take network traces.

[ Edit: I thought I had a working example with headers. I don't; I am blocked either way and mistook the "You're blocked ..." HTML as good data because of course it's compressed. In fact, I am blocked if I use perl and LWP. ]

OK, I pushed away the TODO list (nothing on it I want to do anyway, right?) and did some searching of the web, set up the simplest possible script I could based on someone's suggestion for debugging, and have data on why my requests are being blocked.

All perl examples have been run on OS X with the latest release (i.e. not the version shipped with the OS version I arm running) of perl and LWP:

$ uname -a

Darwin turquoise 15.6.0 Darwin Kernel Version 15.6.0: Mon Oct  2 22:20:08 PDT 2017; rootnu-3248.71.4~1/RELEASE_X86_64 x86_64

$ perl -v

This is perl 5, version 26, subversion 1 (v5.26.1) built for darwin-2level

...

$ perl -MLWP -e 'print $LWP::VERSION, "\n"'

6.29

I've stripped out any input validation or unnecessary error checking, as is traditional for examples.

$ cat lwp-broke

#! /usr/local/bin/perl

use strict;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

my $req = HTTP::Request->new(GET => $ARGV[0]);

my $res = $ua->request($req);

if ($res->is_success) {

    print $res->content;

} else {

    print $res->status_line, "\n";

    print $res->content, "\n";

}

lwp-broke is practically the example from the LWP documentation. It works with example.com, both for HTTP and HTTPS.

$ ./lwp-broke http://www.example.com

<!doctype html>

<html>

<head>

    <title>Example Domain</title>

...

</head>

<body>

<div>

    <h1>Example Domain</h1>

    <p>This domain is established to be used for illustrative examples in documents. You may use this

    domain in examples without prior coordination or asking for permission.</p>

    <p><a href="http://www.iana.org/domains/example">More information...</a></p>

</div>

</body>

</html>

Again, with HTTPS.

$ ./lwp-broke https://www.example.com

<!doctype html>

<html>

<head>

    <title>Example Domain</title>

...

</head>

<body>

<div>

    <h1>Example Domain</h1>

    <p>This domain is established to be used for illustrative examples in documents. You may use this

    domain in examples without prior coordination or asking for permission.</p>

    <p><a href="http://www.iana.org/domains/example">More information...</a></p>

</div>

</body>

</html>

Now the games endpoint, the first one I tried:

$ ./lwp-broke https://api.chess.com/pub/player/skelos/games

403 Forbidden

<!DOCTYPE html>

...

<head>

<title>Access denied | api.chess.com used Cloudflare to restrict access</title>

...

</head>

<body>

...

            <h2 data-translate="what_happened">What happened?</h2>

            <p>The owner of this website (api.chess.com) has banned your access based on your browser's signature (3c4a5f29f9250b5c-ua22).</p>

...

</body>

</html>

$ cat lwp-debug

#! /usr/local/bin/perl

use LWP::UserAgent;

$ua = LWP::UserAgent->new;

$ua->default_header('Accept-Encoding' => scalar HTTP::Message::decodable());

$ua->add_handler("request_send",  sub { shift->dump; return });

$ua->add_handler("response_done", sub { shift->dump; return });

$ua->get($ARGV[0]);

$ ./lwp-debug https://api.chess.com/pub/player/skelos/games

GET https://api.chess.com/pub/player/skelos/games

Accept-Encoding: gzip, x-gzip, deflate, x-bzip2

User-Agent: libwww-perl/6.29

(no content)

HTTP/1.1 403 Forbidden

Cache-Control: max-age=10

Connection: close

Date: Tue, 28 Nov 2017 04:03:53 GMT

Server: cloudflare-nginx

Content-Encoding: gzip

Content-Type: text/html; charset=UTF-8

Expires: Tue, 28 Nov 2017 04:04:03 GMT

CF-RAY: 3c4a70a31ffc9517-NRT

Client-Date: Tue, 28 Nov 2017 04:03:53 GMT

Client-Peer: 104.17.238.85:443

Client-Response-Num: 1

Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2

Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=ssl767067.cloudflaressl.com

Client-SSL-Cipher: ECDHE-ECDSA-AES128-GCM-SHA256

Client-SSL-Socket-Class: IO::Socket::SSL

Client-Transfer-Encoding: chunked

Set-Cookie: __cfduid=d5e68479f6d869afb0aece3d2f58f94b11511841833; expires=Wed, 28-Nov-18 04:03:53 GMT; path=/; domain=.chess.com; HttpOnly

X-Frame-Options: SAMEORIGIN

\37\x8B\10\0\0\0\0\0\0\3\xADWmo\xDB6\20\xFE\xEE_q\xE5\x80\xBE\0\xA1d;\xC9\x92&\x92\x8A"\xCD\x80|Z\xB0%\xD8\x8A\xA20(\xEAd1\x95H\x95\xA4\xACx]\xFF\xFB@Qv$'i\x9Bb\x9F,\xDE\35\x9F\xBB\xE3=\xC7\xA3\xA3g\xEF~?\xBBz\x7Fy\16\x85\xAD\xCAd\22=\xA3\xF4\x83\xC8\xA1\xB4pq\16G\37\23\x88\x9C\2x\xC9\x8C\x89\x89T\xF4\xC6\x80\xC0_A\x95\x99@\2%\x93\xCB\x98\xA0\xA4\xD7\x7F\x92\4\xA2g\37Pf"\xFFH\xE9\35T\x8F\3\xF00\xD4\xD1\xD3\xA0\x8E\xBF\1u\xFC\4\xA8\xA5\xED\xD1\x9C\xE0\xA1,\xEF\xA3P:F*\x90e\xC9$\xB2\xC2\x96\x98\xBC\xE5\34\x8D\x81\14\xA5\xC0\14\xFE\5V\x8B\x80\27hL\xC0U\5\x8D\xC1\14\xCEJ\xD5dy\xC94\x82U\xA0\xD1X-\xB8\5\xD6\xED\x8CB\x8F3\x89*\xB4\14x\xC1\xB4A\33\x93\xEB\xAB\xDF\xE81\x81p\xA3(\xAC\xAD)~n\xC4*&gJZ\x94\x96^\xADk$\xC0\xFD*&\26om\xE8\xF29\xDD\xC2|\13\xE5oz\xFD\x96\x9E\xA9\xAAfV\xA4\xE5\20\xE8\xE2<>\xCF\x96\xB8\xC7\13\xAD*\x8Cg\3\0\xC9*\x8C\x89V\xA9\xB2f\xB0C*!3\xBC\xDD\3\xA9rU\x96\xAA\xBD\xB7e%\xB0\xAD\x95\xB6\x83M\xAD\xC8l\21g\xB8\22\34i\xB7\xD8\23RX\xC1Jj8+1\x9E\xEDU\xECVTM\xB5Y{\xD4R\xC8O\xA0\xB1\x8C\x89\xB1\xEB\22M\x81h\t\x88,&<_x\21\xE5\xC6\20(4\xE61\ty&)_\x8A\xD0\xABB\x9E\7\xA8\xB5\xD2&\xE8\x8C\xEC\xBA\xC6\xFE\xEC\xBAu\x85\x99`11\\#\xCA\xBDZ\xAB\33\xE4V(\xE9}\x8F\xDA\xE4\xF5\xC7\xE4\xF1`^\xDC\5#\xD0\xC5\xF3\xE2\xBB\xF1\10|zHcjv\x90\xBB\xFB\x93Te\xEB/\25\xD3K!O\xA6\xA75\xCB2!\x97'\xD3\xAF\x91\17a\x98\26n\xF22\\\x8B\xDA\16\xA1n\xD8\x8Ay)\1\xA3\xF90\x91Nj\xC2\x9B\xCF\r\xEAuP...

(+ 716 more bytes not shown)

$ ./lwp-debug https://api.chess.com/pub/player/skelos

GET https://api.chess.com/pub/player/skelos

Accept-Encoding: gzip, x-gzip, deflate, x-bzip2

User-Agent: libwww-perl/6.29

(no content)

HTTP/1.1 403 Forbidden

Cache-Control: max-age=10

Connection: close

Date: Tue, 28 Nov 2017 04:06:43 GMT

Server: cloudflare-nginx

Content-Encoding: gzip

Content-Type: text/html; charset=UTF-8

Expires: Tue, 28 Nov 2017 04:06:53 GMT

CF-RAY: 3c4a74c7aafc953b-NRT

Client-Date: Tue, 28 Nov 2017 04:06:43 GMT

Client-Peer: 104.17.237.85:443

Client-Response-Num: 1

Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO ECC Domain Validation Secure Server CA 2

Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=ssl767067.cloudflaressl.com

Client-SSL-Cipher: ECDHE-ECDSA-AES128-GCM-SHA256

Client-SSL-Socket-Class: IO::Socket::SSL

Client-Transfer-Encoding: chunked

Set-Cookie: __cfduid=d40bd8fe3cd6d92cc945f20b493de0da21511842003; expires=Wed, 28-Nov-18 04:06:43 GMT; path=/; domain=.chess.com; HttpOnly

X-Frame-Options: SAMEORIGIN

\37\x8B\10\0\0\0\0\0\0\3\xADWmo\xDB8\14\xFE\x9E_\xC1\xE9\x80\xBD\0\x95\x9D\xA4\xED\xDA\xB5\xB6\x87\xA1\xEB\x80~\xBA\xE2\xAE\xC5\xDD0\14\x81,\xD3\xB1:[\xF2$9nn\xB7\xFF~\x90\xE5\xA4v\xDA\xEE\r\xF7)\26I=$\xC5\x87\xA2\22=y\xFB\xFB\xD9\xD5\xFB\xCBs(lU&\x93\xE8\t\xA5\37D\16\xA5\x85\x8Bs8\xFA\x98@\xE4\24\xC0KfLL\xA4\xA27\6\4\xBE\4Uf\2\t\x94L.c\x82\x92^\xFFI\22\x88\x9E|@\x99\x89\xFC#\xA5wP=\16\xC0\xC3PG?\7u\xFC\r\xA8\xE3\x9F\x80Z\xDA\36\xCD\t\36\xCA\xF2>

\xA5c\xA4\2Y\x96L"+l\x89\xC9\33\xCE\xD1\30\xC8P

\xCC\xE0_`\xB5\10x\x81\xC6\4\\U\xD0\30\xCC\xE0\xACTM\x96\x97L#X\5\32\x8D\xD5\x82[`\xDD\xCE(\xF48\x93\xA8B\xCB\x80\27L\33\xB41\xB9\xBEzG\x8F\t\x84\33EamM\xF1s#V19S\xD2\xA2\xB4\xF4j]#\1\xEEW1\xB1xkC\x97\xCF\xE9\26\xE6[(\x7F\xD3\xEB7\xF4LU5\xB3"-\x87@\27\xE7\xF1y\xB6\xC4=^hUa<\33\0HVaL\xB4J\x955\x83\35R\t\x99\xE1\xED\36H\x95\xAB\xB2T\xED\xBD-+\x81m\xAD\xB4\35ljEf\x8B8\xC3\x95\xE0H\xBB\xC5\x9E\x90\xC2

VR\xC3Y\x89\xF1l\xAFb\xB7\xA2j\xAA\xCD\xDA\xA3\x96B~\2\x8DeL\x8C]\x97h

DK@d1\xE1\xF9\xC2\x8B(7\x86@\xA11\x8FI\xC83I\xF9R\x84^\25\xF2<@\xAD\x956Agd\xD75\xF6g\xD7\xAD+\xCC\4\x8B\x89\xE1\32Q\xEE\xD5Z\xDD \xB7BI\xEF{\xD4&\xAF>&\x8F\7\xF3\xEC.\30\x81.\x9Eg\xDF\x8DG\xE0\xCF\x874\xA6f\7\xB9\xBB?IU\xB6\xFER1\xBD\24\xF2dzZ\xB3,\23ry2\xFD\32\xF9\20\x86i\xE1&/\xC3\xB5\xA8\xED\20\xEA\x86\xAD\x98\x97\x120\x9A\17\23\xE9\xA4&\xBC\xF9\xDC\xA0^\7\x95...

(+ 715 more bytes not shown)

A different client (looks like I have an OS supplied version of curl that's good enough):

$ curl https://api.chess.com/pub/player/skelos/games

{"games":[{"url":"https://www.chess.com/daily/game/172818880","move_by":0, ...

$ curl https://api.chess.com/pub/player/skelos

{"avatar":"https://images.chesscomfiles.com/uploads/v1/user/29738036.9e90e0f7.200x200o.e759af7f32f6.png","player_id":29738036,"@id":"https://api.chess.com/pub/player/skelos","name":"Giles Lean","username":"skelos","followers":48,"country":"https://api.chess.com/pub/country/AU","location":"Melbourne","last_online":1511837490,"joined":1472753335,"status":"premium"}

Apologies for the length of all that; in the end it's more than is necessary I think, but it should be enough for someone else to replicate the problem from a machine which must access chess.com via Cloudflare.

It is, of course, trivially easy to workaround:

$ cat lwp

#! /usr/local/bin/perl

use strict;

use LWP::UserAgent;

my $ua = LWP::UserAgent->new;

$ua->agent("LetMeBeOhGreatCloudflare/1.0");

my $req = HTTP::Request->new(GET => $ARGV[0]);

my $res = $ua->request($req);

if ($res->is_success) {

    print $res->content;

} else {

    print $res->status_line, "\n";

    print $res->content, "\n";

}

$ ./lwp https://api.chess.com/pub/player/skelos

{"avatar":"https://images.chesscomfiles.com/uploads/v1/user/29738036.9e90e0f7.200x200o.e759af7f32f6.png","player_id":29738036,"@id":"https://api.chess.com/pub/player/skelos","name":"Giles Lean","username":"skelos","followers":48,"country":"https://api.chess.com/pub/country/AU","location":"Melbourne","last_online":1511837490,"joined":1472753335,"status":"premium"}