Introducing FlipMind, FlipSight's own Othello AI Engine

Until app v2.0.4, FlipSight used Edax 4.5.6 (okuhara's edax-reversi-AVX) as its engine. It was a great engine in both speed and strength. While building bots for Othello variants such as Anti, Octagon and 10×10, I ended up training and building evaluation functions myself from FlipSight's own game data. That experience made me wonder whether I could build my own engine for standard Othello too, and so I developed the Othello AI engine FlipMind.

How FlipMind was built

• The evaluation functions were trained from data. The variants were trained on FlipSight's own game data, and standard Othello on Egaroucid's training data together with FlipSight's own game data. Egaroucid's data was a huge help in building the standard Othello evaluation function. Many thanks to Takuto Yamana for publishing such a large dataset.
• The search was written from scratch as well. I referred to well-known search techniques, but wrote all the code myself without using code from existing engines. The technical write-ups on Logistello, Edax and Egaroucid, and okuhara's notes on the SSE/AVX optimizations in edax-reversi-AVX, were a great help.
• Results: while keeping the evaluation file at about 14MB, similar to Edax, FlipMind reached an average win rate above 60% against Edax. In BEST mode, which finds the best move, it stays on par with the high-performance Edax AVX version, and computing the score of every move (ALL mode), which the analysis screen relies on, became much faster.
• The web version computes more slowly than the native apps, but it uses the same engine and evaluation function, so there is no difference in strength at the same level.

※ Training data citation: Yamana, Takuto. Egaroucid Free Training Data. https://www.egaroucid.nyanyan.dev/en/technology/train-data/

Benchmarks

FlipMind was compared with two well-known open-source Othello engines, Edax 4.5.6 and Egaroucid 7.8.2.

Speed

The three endgame problem sets below were measured in two modes.

• Edax 16: the 16 endgame positions used by Edax's benchmark command (bench). 20 to 23 empties. Source: Edax bench generator code, the generated 16 positions
• FFO 33-51: FFO endgame test positions #33 to #51 (19 positions). 20 to 27 empties. Official files: fforum-20-39.obf (#33 to #39), fforum-40-59.obf (#40 to #51)
• FFO 40-59: FFO endgame test positions #40 to #59 (20 positions), widely used to compare the endgame performance of Othello engines. 20 to 34 empties, the longest of the three sets. ALL mode was not measured on this set because a single round takes too long. Official file: fforum-40-59.obf

※ The FFO endgame test positions are a collection of endgame problems published on the forum of the French Othello Federation (FFO).

• BEST: finds the best move and its score (used in games). Other moves are only checked to be no better than the best move; their exact scores are not computed.
• ALL: computes the exact score of every legal move (used in analysis).

The analysis screen shows a score for every candidate move, so I judged the speed of ALL mode to be important as well.

Click a time in the tables to see that engine's per-position results and raw output.

x86 AVX2 build (AMD Ryzen 5 5600X)

12 threads, average of 3 rounds per engine.

※ Results can vary with the operating system, CPU, build options and other parts of the environment. This CPU has 6 cores and 12 threads with SMT (Hyper-Threading).

ARM64 build (Apple M4)

FlipSight's iOS and Android apps use this ARM64 build.

MacBook (Apple M4, 10 cores), 10 threads, average of 3 rounds per engine. There are no official macOS executables, so all three engines were built for arm64 from their official sources on this Mac.

Web build (WASM)

Edax has no official WASM build, so FlipSight built and used its own.

Chrome 157, 8 threads for both engines, average of 3 rounds, scores match on every position. Same PC (Ryzen 5 5600X). Multithreaded web builds.

※ Edax has no code for WASM SIMD, so the web used an Edax build without SIMD. FlipMind writes its evaluation code with generic vector types that are not tied to specific CPU instructions, so the same source runs as AVX2 (x86), NEON (ARM) and SIMD128 (web).

Strength

At levels 1, 5, 10, 15 and 20, 2,000 games per level were played from XOT starting positions. Each starting position is played twice, once as Black and once as White, and the disc differences of the two games are added up to count one win, draw or loss (1,000 per level). No opening book was used. The games were played on an AMD Ryzen 5 5600X.

Evaluation file size: FlipMind about 14MB, Edax about 14MB, Egaroucid about 39MB (38MB evaluation + 0.5MB for endgame move ordering).

※ Even at the same level, the detailed settings differ slightly between engines. At levels 1, 5 and 10 all three engines use the same conditions: the midgame is searched to the level's depth at 100%, and the endgame is solved exactly from 2 × level empties. At levels 15 and 20, the confidence used when reading the late phase to the end differs between engines. Higher confidence takes more time but is more accurate.

Level 15

Level 20

※ The average time per game is the search time each engine spent in a game, averaged over the 2,000 strength games. It is the search time reported by each engine itself and does not include command handling or I/O. Several games were played at the same time, so please read these numbers as a comparison between engines rather than as absolute values.

Availability

• Web: FlipMind already powers FlipSight's engine features, including analysis and evaluation.
• iOS and Android apps: FlipMind is used from v2.0.5. Please update the app to v2.0.5 or later.

Since the core engine has changed, there may be unexpected problems. If you notice anything wrong, please let me know by message, email or by sending a diagnostic report. I will check and fix it quickly.