When Haidilao Hotpot Maze Meets Image Processing

Image Processing, Computer Vision and AI + IoT + 5G are going to and have been changing our life. They should not be something mysterious but something be applied to daily life.

English

中文啦

This is an article inspired by a free side dish + my curious soul.

I will divide my article into 2 sections: the fun part and the technical part.

The Fun Part will show you what this project does and how it looks like. That's it, simple and short :) but if you were as curious as me and would like to know the details of the implementation then keep reading.

In the Technical Part, I will treat it as my technical note to remind me what I did for solving the problems and why I did that.

The Fun Part

It was a Sunday afternoon, we had been cooking for ages and were sick of doing so... finally we decided to try the expensive hotpot Haidilao. The Haidilao is famous about its premium services and expensive cost.

I know they offer nail and hand skin-care services to customers in waiting queue and yes I actually got the service :)

我的目的很单纯，就是想那一盘免费的菜～ 没想到能扯这么多出来～

我会把本文分成两个部分： 看热闹 和 从技术角度谈如何实现这个小项目。

看热闹部分会展示这个项目看起来长什么样，能做什么。就是这么短小精悍。但如果你和我一样好奇 想它怎么实现的，请继续阅读技术部分。

技术部分对我来说是相当于笔记，记录下来我做了什么来解决这个问题，还有我为什么这么做。这里不讨论具体如何做。

一个风和日丽的周日， 在家做饭许久的我们，被困在澳洲不能回国已一年有余，生活是如此单调。。。最终我们决定去吃饭，试一试听说了许久但从来没舍得吃的 海～底～捞。都知道海底捞服务一流，排队送美甲服务、手部护理。我也去试一试～！

nail service

But what I did not know was that they also offered me a little maze to play to kill the waiting time. What's even better was if I got it solved while I was still in the waiting queue, they would reward me with a free dish 😊!

maze

到了以后 果然如此！但我不知道的是，排队等候的时候我们还拿到了两张 小纸片，上面印有迷宫。解出来就有菜吃～！ 嘿嘿黑，这不是送分题吗～太简单了～

不到10分钟， 我就被莉莉打败了。。。。好吧 咳咳，如果我不能打败你， 我就编个程序来战胜你。 从照片中提取 迷宫，看起来很有趣～！而且这正好是计算机图像处理的领域，也是我的最爱～

我们先看来来整个程序是怎样的。

A piece of cake~!

Within 10 minutes, Lily beat me.... huh.... okay, if I couldn't beat you, I could make a program to beat you. Extracting a maze from a photo looks like a good fit for Image Processing!

Here is what it looks like~!

简单介绍一下： 点击猪哥亮 选择你拍摄的 迷宫照片。（小窍门： 保持迷宫在照片的正中间，确保迷宫的外壳（不是那条它外面的壳）是最外面一层的。让迷宫纸片尽量摊平，和背景区分开来。）剩下的就交给Haidilao Maze Buster来做吧～

如果你也碰巧要去吃海底捞，下面就是这个app的网站，可以试一试，但不保证管用 :)

As you can see here, the app takes a photo and extracts the maze inside the photo and solves the maze problem.

If you were going to have your lovely Haidilao Hotpot, then I have also made it available for you :)


>>>Haidilao Maze Buster<<<



技术部分

视频看起来挺短，做解迷宫的部分也就花了2、3天， 但是为了让它更稳定，并且找到合适的免费hosting服务 我找了估计至少有3-4周。。。。

现在它仍有一些不足，比如说：如果箭头堵住迷宫出入口的话 就无法识别出进出口。不过我已经很满意当前的表现了， 毕竟我还有很多其他事情要做， 没法投入太多时间。具体有哪些限制，下面会提到。

是我怎样实现它的 大体步骤：

👉 从照片中提取出迷宫并且让迷宫正直的面对我们

👉 找到出入口

👉 找出迷宫路经

接下来 具体阐述如果实现迷宫提取和 矫正迷宫

这是细分的步骤：

  1. 对照片做白平衡
  2. 标出低饱和度的区域（因为 碰巧发现迷宫就在这个区域）
  3. 运用 Morphology Close/Open来擦掉迷宫里面的具体信息 只留下一片空白区域
  4. 找到图中最外面一层的轮廓（这是其中的一个限制）
  5. 裁减掉除迷宫外的其他区域（这个时候迷宫纸还在）这样可以减少接下来的计算量
  6. 二值化图像，让图像非黑(0)即白(255) 来进一步减少运算量 并且也有助于下一步
  7. 从图像中间的一部分区域获取种子图像(这就是为什么 迷宫必须在照片中间）然后运用Morphological Reconstruction来提取迷宫（这个时候迷宫纸就没有了，只剩下纯净的迷宫，但纸所占的区域还在裁减后的图像里）
  8. 颠倒黑白像素值
  9. 获取迷宫的 凸包
  10. 使用获得的凸包找到它的4个角
  11. 再次 裁减 迷宫，整个图就只剩迷宫
  12. 如果迷宫角度不正，旋转迷宫让它摆正位置

值得指出的是白平衡真太重要啦～！我感觉我浪费了2周的时间就是因为没有做这一步。 图像处理的白平衡，我自己的感觉是，就像 AI里面的 Normalization 一样。如果不做这一步，同样的代码，这张图可以，灯光稍微改变一些 就会导致另一张图就不行。

为什么这么说：因为如果灯光的亮度，温度，等变化了（对我们人眼都没影响）但是对于计算机来说就是完全不一样了。

来看一下白平衡后的照片和原照片的对比

The technical part

The demo video looks pretty short, to implement it from the scratch, it only took me 2 or 3 days, but... to perfect it and make it available online actually cost me 3 or 4 weeks....

After couple of weeks working on it, it's still not perfect but good enough to satisfy my requirements, so don't expect it to be a commercial project which works for each and every scenario, I have to admit that it has its limitations. I will explain what the limitations are in the following paragraphs.

So here is the procedure how I got it implemented:

👉 Extract the maze out of the photo and warp/rotate it to face directly to us

👉 Find its 2 entrances, the Start and the Exit

👉 Find the path :)

Now, let's talk about how to extract the maze out of the photo and get it face straight to us

There are certain steps to do:

  1. Get the white balanced image first~!!
  2. Mask the low saturation area, because the maze area happens to be the low saturation area,
  3. Use Morphology Close/Open to remove details of the maze so that we only get a big block of white area where the maze is located,
  4. Find the most outside contour of the image(now you know one of the limitations),
  5. Crop the image so that it only contains the maze area and thus reduces the computation later on,
  6. Threshold the image, so that only 0 and 255 are left to the values of pixels, it simplifies the following processing,
  7. Use a seed image which is created by taking the center part of the maze image and apply Morphological Reconstruction to extract the maze,
  8. Invert the extracted maze to be white wall and black path,
  9. Get the convex hull of the extracted maze,
  10. Use the convex hull to get its corners,
  11. Then crop the image again to extract the exact maze,
  12. Warp the maze if it's rotated and get it ready for the next step.

I cannot stress it enough that how important of white balance is. I can tell you that 2 weeks of my precious time was wasted because I did not do the White Balance. To my understanding, it behaves like Normalization in Computer Vision for AI. Without doing it, your code would be fragile and endless frustration is waiting ahead.

What did I mean by this?

Any changes of lighting, the human eyes cannot event notice the differences, will jeopardize the result and breaks the program. I was scratching my head to try to figure out why it works for one image but not for another.

To give you an idea of how the white balanced image look like:

As you can see above, it does change the look and feel of the image by equalizing the Red, Green and Blue channels.

This is very helpful, as it makes the rest of the code free from being affected by any changes of lighting, so the code is more stable.

I wasn't thinking of using HSV color space at first as RGB is the most popular color space. But by operating in the HSV, I found out that the maze area is always the darkest area.

可以看出，图片的温度好像改变了，这是通过 均衡化 红绿蓝三色通道达到的。 这一步非常重要，它能让接下来的代码不会受到任何由于光改变所带来的影响，所以代码就会更加可靠一些。 我起先并没有想到用HSV颜色空间，因为大部分时间都是在RGB空间下工作。 但通过观察HSV空间下的 迷宫，我发现 迷宫区域总是最暗的一部分。

This is great, as it enables me to easily pick the maze paper out of the whole photo(Mask).

As I mentioned above, by also applying morphological close and open I can erase out the rest image and leave a big block of white area to indicate where the maze is located:

这就太棒啦！因为这样我就能轻松的提取出迷宫部分而不需要做各种复杂的图像处理（图Mask)。之后再用Morphology Close/Open（MC/MO）来擦除迷宫内部细碎的细节和周边的噪音点。