Personal Log - Adding 'Draw' Feature
This article, completely original, is copyrighted by its author, me. Please do not reproduce it.
本文为原创作品，作者 Kolyn090 拥有其著作权，受法律保护。严禁复制、转载、仿冒或以任何形式使用。
Category: Log
Prerequisite: Basic Python
Introduction
Nonogram Solver is a puzzle solver for Nonogram I made a while ago. Its algorithm is based on fedimser’s nonolab. Its UI code is based on another project of mine, you can find it on Linkedin.
Nonogram Solver currently supports solving Nonogram puzzles, importing puzzles, and exporting puzzles. Most importantly, importing puzzles from images. Today I will be working on ‘Drawing a Puzzle’ feature.
Start working
Step 1
The first thing I am going to do is to review my current project. Here is the project hierarchy of my current project:
src
|_ image_recognition
|_ solve
|_ ui
|_ utilHere, ui contains all front end code. image_recognition takes in a screenshot and attempts to find the puzzle (it’s actually just two matrices). solve solves the given puzzle if it’s valid. util contains all utility functions.
Step 2
I like to separate things this way to keep them stay organized. Now let’s think how ‘Draw a Puzzle’ should fit into this system. First of all, I think it deserves its own folder.
src
|_ draw
|_ image_recognition
|_ solve
|_ ui
|_ utilAlso, since its functionality is drawing, there will be some connections with the UI components. So this time I cannot treat it the same as solve nor image_recognition. Create draw_mode under ui.
src
|_ draw
|_ image_recognition
|_ solve
|_ ui
|_ draw_mode
|_ utilStep 3
Think about this, the problem can be simplified as given a binary image, return a Nonogram puzzle (two matrices). So the next thing I should do is actually quite easy: Take a binary image and try to convert it to a more familiar data.
Let’s see this example:
For now, let me convert this to ASCII Art. Create binary_to_ascii_art.py under draw.
import cv2
from src.solve.solution import EMPTY_MARKER, FILL_MARKER
class Binary_To_Ascii_Art:
def __init__(self, binary_image):
self.EMPTY_MARKER = EMPTY_MARKER
self.FILL_MARKER = FILL_MARKER
height, width = binary_image.shape
self.ascii = []
# Iterate through each pixel
for y in range(height): # Rows
row = []
for x in range(width): # Columns
pixel_value = binary_image[y, x]
# Check if the pixel is black or white
if pixel_value == 255:
row.append(EMPTY_MARKER)
elif pixel_value == 0:
row.append(FILL_MARKER)
self.ascii.append(row)
if __name__ == '__main__':
binary_image = cv2.imread('binary.png', cv2.THRESH_BINARY)
btaa = Binary_To_Ascii_Art(binary_image)
for lst in btaa.ascii:
print(''.join(lst))I got
☐☐☐☒☒☒☒☒☒☐☐☐☐☐
☐☐☒☐☐☐☐☐☐☒☐☐☐☐
☐☒☐☒☒☒☒☒☒☐☒☐☐☐
☐☒☒☒☒☒☒☒☒☒☐☒☐☐
☐☒☒☒☒☐☒☒☒☒☒☐☒☐
☐☒☒☒☒☐☒☒☒☒☒☒☒☒
☒☐☐☐☒☒☒☒☒☒☒☒☒☒
☒☒☒☐☒☒☒☒☒☒☒☒☒☒
☐☒☒☒☒☒☒☒☒☒☒☒☒☒
☐☐☒☐☒☐☒☒☒☒☒☒☒☒
☐☐☒☒☒☒☒☒☒☒☒☒☒☒
☐☐☐☒☒☒☒☒☒☒☒☒☒☐
☐☐☐☐☒☐☒☒☒☐☒☒☐☐
☐☐☐☐☐☒☒☒☒☒☒☐☐☐Step 4
I know nonolab has drawing feature. It takes in an ASCII Art like above and returns a .non file, which is the file format that nonolab uses to save a Nonogram puzzle.
Save the ASCII Art into talpia.txt and run Main.java. Type create talpia.txt, I got
width 14
height 14
columns
2
4,2
1,3,4
1,4,1,2
1,11
1,2,3,2,1
1,12
1,12
1,12
1,9,1
1,10
1,8
8
6
rows
6
1,1
1,6,1
9,1
4,5,1
4,8
1,10
3,10
13
1,1,8
12
10
1,3,2
6
goal "0000001100000000111101100000010111011110001011110010110010111111111110101100111011011011111111111110111111111111101111111111110101111111110100101111111111000101111111100000111111110000000111111000"Now I import this file into my Nonogram Solver, I got
Step 5
Next, I want to analyze fedimser’s code. In Main.java, I found
private static void create(String fileName) {
...
String ascii = Files.lines(file.toPath()).collect(Collectors.joining("\n"));
sol = new NonogramSolution(ascii);
...
}In NonogramSolution.java, this constructor converts a given ASCII Art into a 2D array of booleans. With black pixel being true and white pixels being false.
public NonogramSolution(String picture) throws IllegalArgumentException {
...
}In NonogramSolution.java
public List<Integer> getRowDescription(int y) {
List<Integer> ans = new ArrayList<Integer>();
int cum = 0;
for (int x=0;x<width;x++) {
if (pixels[x][y]) cum++;
else if (cum > 0) {ans.add(cum); cum = 0;}
}
if (cum > 0) ans.add(cum);
return ans;
}
public List<Integer> getColumnDescription(int x) {
List<Integer> ans = new ArrayList<Integer>();
int cum = 0;
for (int y=0;y<height;y++) {
if (pixels[x][y]) cum++;
else if (cum > 0) {ans.add(cum); cum = 0;}
}
if (cum > 0) ans.add(cum);
return ans;
}Given the hint, now I am also going to convert the binary image into a matrix of booleans.
self.ascii = []
self.pixels = []
# Iterate through each pixel
for y in range(height): # Rows
row = []
pixel_row = []
for x in range(width): # Columns
pixel_value = binary_image[y, x]
# Check if the pixel is black or white
if pixel_value == 255:
row.append(EMPTY_MARKER)
pixel_row.append(False)
elif pixel_value == 0:
row.append(FILL_MARKER)
pixel_row.append(True)
self.ascii.append(row)
self.pixels.append(pixel_row)
for lst in btaa.pixels:
print(''.join(str(lst)))I got
[False, False, False, True, True, True, True, True, True, False, False, False, False, False]
[False, False, True, False, False, False, False, False, False, True, False, False, False, False]
[False, True, False, True, True, True, True, True, True, False, True, False, False, False]
[False, True, True, True, True, True, True, True, True, True, False, True, False, False]
[False, True, True, True, True, False, True, True, True, True, True, False, True, False]
[False, True, True, True, True, False, True, True, True, True, True, True, True, True]
[True, False, False, False, True, True, True, True, True, True, True, True, True, True]
[True, True, True, False, True, True, True, True, True, True, True, True, True, True]
[False, True, True, True, True, True, True, True, True, True, True, True, True, True]
[False, False, True, False, True, False, True, True, True, True, True, True, True, True]
[False, False, True, True, True, True, True, True, True, True, True, True, True, True]
[False, False, False, True, True, True, True, True, True, True, True, True, True, False]
[False, False, False, False, True, False, True, True, True, False, True, True, False, False]
[False, False, False, False, False, True, True, True, True, True, True, False, False, False]Now write a class to convert pixels to description. Create pixels_to_description.py under draw.
import cv2
from src.solve.description import Description
from src.draw.binary_to_ascii_art import Binary_To_Ascii_Art
class Pixels_To_Description:
def __init__(self, pixels):
self.description = Description()
self.description.width = len(pixels)
self.description.height = len(pixels[0])
self.pixels = pixels
self.description.row_descriptions = [self.get_row_description(y) for y in range(self.description.height)]
self.description.column_descriptions = [self.get_col_description(x) for x in range(self.description.width)]
def get_row_description(self, y):
result = []
cum = 0
for x in range(self.description.width):
if self.pixels[x][y]:
cum += 1
elif cum > 0:
result.append(cum)
cum = 0
if cum > 0:
result.append(cum)
return result
def get_col_description(self, x):
result = []
cum = 0
for y in range(self.description.height):
if self.pixels[x][y]:
cum += 1
elif cum > 0:
result.append(cum)
cum = 0
if cum > 0:
result.append(cum)
return result
if __name__ == '__main__':
binary_image = cv2.imread('binary.png', cv2.THRESH_BINARY)
btaa = Binary_To_Ascii_Art(binary_image)
ptd = Pixels_To_Description(btaa.pixels)
description = ptd.description
print(description.row_descriptions)
print(description.column_descriptions)I got the same result. This is what I called ‘the two matrices’ to form the puzzle.
[[2], [4, 2], [1, 3, 4], [1, 4, 1, 2], [1, 11], [1, 2, 3, 2, 1], [1, 12], [1, 12], [1, 12], [1, 9, 1], [1, 10], [1, 8], [8], [6]]
[[6], [1, 1], [1, 6, 1], [9, 1], [4, 5, 1], [4, 8], [1, 10], [3, 10], [13], [1, 1, 8], [12], [10], [1, 3, 2], [6]]Step 6
Now I should design the ‘Draw’ feature in UI. Here is my app’s current UI layout.
The plan:
- Add a new button
Draw - During
Draw,Solvebutton will be disabled Resetstill works the sameExportandImportbuttons will be disabledExperimentalbutton will be disabled- I will need a
Finishbutton in draw mode. After pressed, put the description in the entries. - Press
Drawbutton again to exit draw mode
Therefore, I need Draw, Reset, and Finish buttons to be visible in draw mode.
Add more code in ui.py under ui to achieve this.
def __init__(self, master=None, **kwargs):
...
self.draw_text = "Draw"
self.stop_draw_text = "Stop draw"
self.draw_button = tk.Button(button_frame, text=self.draw_text, command=self.press_draw_button)
self.draw_button.grid(row=5, column=0)
self.finish_button = tk.Button(button_frame, text="Finish", command=self.finish_draw)
self.finish_button.grid(row=6, column=0)
self.finish_button.config(state=tk.DISABLED)
self.default_buttons = [solve_button, reset_button, export_button,
import_button, experimental_button, self.draw_button]
self.draw_mode_buttons = [reset_button, self.draw_button, self.finish_button]
def press_draw_button(self):
def enter_draw_mode():
self.draw_mode = True
self.draw_button.config(text=self.stop_draw_text)
for i in range(len(self.default_buttons)):
self.default_buttons[i].config(state=tk.DISABLED)
for i in range(len(self.draw_mode_buttons)):
self.draw_mode_buttons[i].config(state=tk.NORMAL)
def exit_draw_mode():
self.draw_mode = False
self.draw_button.config(text=self.draw_text)
for i in range(len(self.draw_mode_buttons)):
self.draw_mode_buttons[i].config(state=tk.DISABLED)
for i in range(len(self.default_buttons)):
self.default_buttons[i].config(state=tk.NORMAL)
if not self.draw_mode:
enter_draw_mode()
else:
exit_draw_mode()
def finish_draw(self):
print('finish')The buttons in default mode:
The buttons in draw mode:
Step 7
Now the next step will be drawing binary images. Create draw_mode.py under draw_mode. This class is going to manage draw mode as an additional feature of paintboard.py. For debugging purpose, let me save the drawn image for now.
import cv2
import numpy as np
class Draw_Mode:
def __init__(self, paintboard):
self.paintboard = paintboard
self.pixel_size = paintboard.pixel_size
self._draw_mode = False
def handle_click(self, event=None):
if not self._draw_mode:
return
# Calculate the pixel location based on click coordinates
row = event.x // self.pixel_size
col = event.y // self.pixel_size
if 0 <= row < self.paintboard.grid_width and 0 <= col < self.paintboard.grid_height:
self.paintboard.pixels[row, col] = self.paintboard.paint_rgb
self.paintboard.paint_pixel(row, col)
def get_binary_image(self):
def make_binary(cv2_img, threshold=127):
# Convert to grayscale
grayscale = cv2.cvtColor(cv2_img, cv2.COLOR_BGR2GRAY)
# Apply binary threshold
_, binary = cv2.threshold(grayscale, threshold, 255, cv2.THRESH_BINARY)
return binary
cv2.imwrite('drawing.png', np.flip(np.rot90(make_binary(self.paintboard.pixels), k=1), axis=0))
def start_draw_mode(self):
self._draw_mode = True
self.paintboard.reset()
def end_draw_mode(self):
self._draw_mode = False
self.paintboard.reset()In paintboard.py
def __init__():
self.draw_mode = Draw_Mode(self)
self.canvas.bind('<Button-1>', self.draw_mode.handle_click)
self.canvas.pack()
def start_draw_mode(self):
self.draw_mode.start_draw_mode()
def end_draw_mode(self):
self.draw_mode.end_draw_mode()
def save_binary_image(self):
self.draw_mode.get_binary_image()
def reset(self):
for row in range(len(self.pixel_ids)):
for col in range(len(self.pixel_ids[row])):
if self.pixel_ids[row][col] is not None:
self.canvas.delete(self.pixel_ids[row][col])
self.pixels = np.full([self.grid_width, self.grid_height, 3], 255, dtype=np.uint8)
self.picture = None
self.canvas.bind('<Button-1>', self.draw_mode.handle_click)
self.canvas.pack()Modify ui.py.
def press_draw_button(self):
def enter_draw_mode():
self.draw_mode = True
self.paintboard.start_draw_mode()
self.draw_button.config(text=self.stop_draw_text)
for i in range(len(self.default_buttons)):
self.default_buttons[i].config(state=tk.DISABLED)
for i in range(len(self.draw_mode_buttons)):
self.draw_mode_buttons[i].config(state=tk.NORMAL)
def exit_draw_mode():
self.draw_mode = False
self.paintboard.end_draw_mode()
self.draw_button.config(text=self.draw_text)
for i in range(len(self.draw_mode_buttons)):
self.draw_mode_buttons[i].config(state=tk.DISABLED)
for i in range(len(self.default_buttons)):
self.default_buttons[i].config(state=tk.NORMAL)
...
def finish_draw(self):
self.paintboard.save_binary_image()After drawing and saving, I got
Step 8
This is good, but I think it’s nicer to be able to erase. So that would be the next thing I implement.
This can be easily achieved by checking the current color of the pixel. If it’s black, paint it white, otherwise black. Actually, it’s a little more than that in my code: if it’s white, remove the rectangle and don’t insert new one, otherwise insert a black one.
In draw_mode.py
def handle_click(self, event=None):
...
if np.array_equal(curr_pixel, self.paintboard.paint_rgb):
self.paintboard.pixels[row][col] = self.paintboard.default_pixel_rgb
else:
self.paintboard.pixels[row][col] = self.paintboard.paint_rgb
...In paintboard.py
def paint_pixel(self, row, col):
...
if not np.array_equal(self.pixels[row][col], self.default_pixel_rgb):
color = self.rgb_to_hex(self.pixels[row][col])
else:
color = None
...
if color:
# Draw the rectangle (pixel) and store its ID for future reference
self.pixel_ids[row][col] = self.canvas.create_rectangle(x1, y1, x2, y2, fill=color, outline=color)🎉 Now we have got an eraser!
Step 9
Currently, pressing ‘Finish’ button will only save the binary image. However, the goal is to display the puzzle entries.
Back in step 5, Pixels_To_Description is already done.
In ui.py
def finish_draw(self):
binary_image = self.paintboard.get_binary_image()
btaa = Binary_To_Ascii_Art(binary_image)
ptd = Pixels_To_Description(btaa.pixels)
description = ptd.description
self.rows.load(description.row_descriptions)
self.cols.load(description.column_descriptions)
self.paintboard.render_picture(btaa.pixels)In paintboard.py, I changed save_binary_image to get_binary_image.
def get_binary_image(self):
return self.draw_mode.get_binary_image()I also found something I am unsure of. It’s unclear to me why the program only work without these transformations. They were required for saving the array as an image.
def get_binary_image(self):
...
# image = np.flip(np.rot90(make_binary(self.paintboard.pixels), k=1), axis=0)
image = make_binary(self.paintboard.pixels)
# cv2.imwrite('drawing.png', image)
return imageAfter everything’s done. The app can finally draw puzzles. Here is the final result.
If you liked this log, consider giving a Star to this repository.
🍯 Happy Coding 🍯
This article, completely original, is copyrighted by its author, me. Please do not reproduce it.
本文为原创作品，作者 Kolyn090 拥有其著作权，受法律保护。严禁复制、转载、仿冒或以任何形式使用。