DeepMind Neural Network Strength

Check out chess.com's latest article on the topic (link below) where it's proven beyond any doubt that...

that while amazing, the chess playing strength of neural networks is completely overblown.

In the article it's reported that against an old version of stockfish without opening book, it scored 57% which corresponds to an Elo of 60 points higher.

When SF was given an opening book that fell to only 54% (they don't mention this in the article, you have to find it yourself). This corresponds to an Elo of only 30 points higher against an old version of stockfish.

So lets see a current version on strong hardware with long time controls with an opening book and access to EGTB. Even if the NN could win, it'd be more like 50.5% to 49.5% which of course doesn't make good headlines so it will never happen.

What are your thoughts?

I already think NNs are an amazing piece of engineering... what I'm calling BS on is their incredible chess strength, which is completely overblown... but I'm willing to listen to arguments otherwise.

https://www.chess.com/news/view/updated-alphazero-crushes-stockfish-in-new-1-000-game-match

In this way we can estimate that current SF is rated roughly 80 point higher than the bunk being hyped in the NN article.

Alphazero is just a stunt but we should remember it was the first of its kind, however, NNs (and really only Lc0 and Leelenstein) have shown to give a run for its money to SF as the CCC12 shows. I've always wanted to see a Brainfish (SF with Cerebellum opening book) against Lc0

(first, a disclaimer-i have never studied a damn thing when it comes to chess)

wow! my first instinct is to ask how all the 64 bit and cpu stuff matters (and, i do not understand much of the first graph without the header. then again, i might not understand it with the header). 

my second instinct is thinking that many chess engines can show best moves, but regular janes and joes will still not see why. 

@AlCzervik

 - Simply put 64 bit is faster. It can process more information.

 - I don't understand all those columns of numbers. The important piece is just the rating anyway.

 - Drmrboss informed me in a different topic that a NN did play one of the latests SFs and the match means it was rated about 25 points higher than SF.

 - re: humans wont understand so why does it matter. A tweet by Kasparov said it's nice that these NN engines are playing open active games instead of incomprehensible maneuvering (not that we understand every move, but it's at least recognizable).

 - I agree though that it's more a feat of technology at this point, because certainly humans can't tell the difference between 3200 and 3400... which is partly why I internally laugh when someone proclaims ____ is the "best" move. A new engine could disagree tomorrow and we wouldn't know the difference. Another reason is engines are so much better that their suggestions are sometimes (often?) impractical for us to play.

Drmrboss is talking about the CCC12 event:

https://www.chess.com/computer-chess-championship#event=ccc1216-lc0-vs-stockfish

Thanks

It is difficut to determine whether CPU engine ( Stockfish) is better or NN engine (Leela ) is better because they run on completely different hardwares.

If you give 128 cores cluster cpu to Stockfish and  1 RTX GPU to Leela  , Stockfish will smash leela but 1 core CPU SF will badly lose to 1 RTX GPU.

 Currently , 16 cores CPU Stockfish ($800 for CPU only) will equally perform to one 2080 RTX Ti GPU ( $1000 for GPU only).

With current technology and cheaper prize, 2x 2080TI ( $2000 for GPU only) on Leela can run similar performance as  Alpha Zero running on 4 TPU in 2017. And also 64 thread AMD 3970X ($2000) can run similar performane on Stockfish 8 hardware that deep mind used.

As a consumer point of view,  you can decide whether you can afford to upgrade CPU or GPU or both.

Oh look, I predicted your ability to misunderstand my topic, how... completely expected.

From the OP:

Hah, ok, so people who make good use of software and hardware are not yet impressed... and certainly not to the extent articles (as on chess.com) pretend we should be.

It is possible to run Leela and Stockfish on the same kit. I've tried it on my own PC with the no GPU version of Leela.

I was interested to see if Leela could play basic endgames any better than SF8 (without EGTBs).

The answer is, at G120+10, SF8 is light years ahead of Leela on these endgames. I also tried the top level on lichess, which is apparently SF10 and responds in a few seconds. That was in turn light years ahead of SF8 at the longer time control on my PC.

So it would seem SF wins, at least on basic endgames, unless you give Leela a few thousand extra processors (in the form of a GPU).

None of the programs could play KNNKP, but I suspect SF10 would if it were given the same resources I gave the others (I haven't actually tried it yet). SF8 couldn't play it a lot better than Leela couldn't and similarly the time limited SF10 on lichess a lot better than SF8 on my PC.  Leela (only) couldn't play KBNK.

Download LD2 or Latest T59 version of Leela. They are at the Strength of Stockfish 7.

Even CPU Leela is top 5 ( at Ethereal Engine Strength) after Stockfish, komodo, houdini and fire.

In fact running Leela on cpu is more idiot than a grandma driving formula 1 in muddy lane and saying " Oh, formula 1 is slower than my carts driven by two bulls.

But isn't that the point?

If you want to test the relative talents of two racing drivers you don't put one in a formula 1 car and the other in a bull cart. 

I ran the two algorithms on exactly the same kit. Leela was useless.

I know that if I invested in a graphics card Leela would do better, but then it would be implemented on a couple of thousand processors in comparison with Stockfish's four. Formula 1 vs. bullcart.

You said:

It is difficut to determine whether CPU engine ( Stockfish) is better or NN engine (Leela ) is better because they run on completely different hardwares.

The answer is to run them on the same hardware as I did.

It does not matter. Thw thing is that NN learn chess by itself (including the opening theory) , that's the amazing thing about them. They also have a very attractive playing style, going for dynamic attacking positions.

I'm suitably impressed, but it does seem that the grunt required to handle the NN is too much to give reasonable results on a basic PC.

So you're obviously making use of your knowledge. Long term considerations that engines can't see, but I've never played a computer correspondence game like that. I've seen positions where the engine completely misunderstands an attack, but that's about it.

Do you have any fun examples of wins against clueless engine monkeys? I'd be interested in seeing any you're willing to share.

My guess is that engines aren't very good at the opening/early midgame, how do I know? Because NNs (not only leela but even bad engines like Stoofvlees) constantly get very nice positions after that phase and SF is often having to struggle for a draw. Engines are also famous for misevaluating fortresses, even if they can find the proper line, they often get into endings that look winning but in fact aren't.

What I think is terrible about engines is you’re losing your entire focus that you should be working to improve, while moving and finding the proper placement each piece.

Using those has you more on such reliance, where relying on your own positioning, that you’re mindset is that you will find your proper placement each move, should be one’s main standpoint, progressing more and more by doing so and actually playing chess and not directed where best is your move, that you apparently (because how programmed), think using engines are wise stumps your increased skills the game.

That's probably good advice