]>
Commit | Line | Data |
---|---|---|
5a27c734 MY |
1 | #!/usr/bin/env python2 |
2 | # | |
3 | # Author: Masahiro Yamada <[email protected]> | |
4 | # | |
5 | # SPDX-License-Identifier: GPL-2.0+ | |
6 | # | |
7 | ||
8 | """ | |
9 | Move config options from headers to defconfig files. | |
10 | ||
11 | Since Kconfig was introduced to U-Boot, we have worked on moving | |
12 | config options from headers to Kconfig (defconfig). | |
13 | ||
14 | This tool intends to help this tremendous work. | |
15 | ||
16 | ||
17 | Usage | |
18 | ----- | |
19 | ||
20 | This tool takes one input file. (let's say 'recipe' file here.) | |
21 | The recipe describes the list of config options you want to move. | |
22 | Each line takes the form: | |
23 | <config_name> <type> <default> | |
24 | (the fields must be separated with whitespaces.) | |
25 | ||
26 | <config_name> is the name of config option. | |
27 | ||
28 | <type> is the type of the option. It must be one of bool, tristate, | |
29 | string, int, and hex. | |
30 | ||
31 | <default> is the default value of the option. It must be appropriate | |
32 | value corresponding to the option type. It must be either y or n for | |
33 | the bool type. Tristate options can also take m (although U-Boot has | |
34 | not supported the module feature). | |
35 | ||
36 | You can add two or more lines in the recipe file, so you can move | |
37 | multiple options at once. | |
38 | ||
39 | Let's say, for example, you want to move CONFIG_CMD_USB and | |
40 | CONFIG_SYS_TEXT_BASE. | |
41 | ||
42 | The type should be bool, hex, respectively. So, the recipe file | |
43 | should look like this: | |
44 | ||
45 | $ cat recipe | |
46 | CONFIG_CMD_USB bool n | |
47 | CONFIG_SYS_TEXT_BASE hex 0x00000000 | |
48 | ||
96464bad JH |
49 | Next you must edit the Kconfig to add the menu entries for the configs |
50 | you are moving. | |
51 | ||
5a27c734 MY |
52 | And then run this tool giving the file name of the recipe |
53 | ||
54 | $ tools/moveconfig.py recipe | |
55 | ||
56 | The tool walks through all the defconfig files to move the config | |
57 | options specified by the recipe file. | |
58 | ||
59 | The log is also displayed on the terminal. | |
60 | ||
61 | Each line is printed in the format | |
62 | <defconfig_name> : <action> | |
63 | ||
64 | <defconfig_name> is the name of the defconfig | |
65 | (without the suffix _defconfig). | |
66 | ||
67 | <action> shows what the tool did for that defconfig. | |
68 | It looks like one of the followings: | |
69 | ||
70 | - Move 'CONFIG_... ' | |
71 | This config option was moved to the defconfig | |
72 | ||
73 | - Default value 'CONFIG_...'. Do nothing. | |
74 | The value of this option is the same as default. | |
75 | We do not have to add it to the defconfig. | |
76 | ||
77 | - 'CONFIG_...' already exists in Kconfig. Do nothing. | |
78 | This config option is already defined in Kconfig. | |
79 | We do not need/want to touch it. | |
80 | ||
81 | - Undefined. Do nothing. | |
82 | This config option was not found in the config header. | |
83 | Nothing to do. | |
84 | ||
85 | - Failed to process. Skip. | |
86 | An error occurred during processing this defconfig. Skipped. | |
87 | (If -e option is passed, the tool exits immediately on error.) | |
88 | ||
89 | Finally, you will be asked, Clean up headers? [y/n]: | |
90 | ||
91 | If you say 'y' here, the unnecessary config defines are removed | |
92 | from the config headers (include/configs/*.h). | |
93 | It just uses the regex method, so you should not rely on it. | |
94 | Just in case, please do 'git diff' to see what happened. | |
95 | ||
96 | ||
97 | How does it works? | |
98 | ------------------ | |
99 | ||
100 | This tool runs configuration and builds include/autoconf.mk for every | |
101 | defconfig. The config options defined in Kconfig appear in the .config | |
102 | file (unless they are hidden because of unmet dependency.) | |
103 | On the other hand, the config options defined by board headers are seen | |
104 | in include/autoconf.mk. The tool looks for the specified options in both | |
105 | of them to decide the appropriate action for the options. If the option | |
106 | is found in the .config or the value is the same as the specified default, | |
107 | the option does not need to be touched. If the option is found in | |
108 | include/autoconf.mk, but not in the .config, and the value is different | |
109 | from the default, the tools adds the option to the defconfig. | |
110 | ||
111 | For faster processing, this tool handles multi-threading. It creates | |
112 | separate build directories where the out-of-tree build is run. The | |
113 | temporary build directories are automatically created and deleted as | |
114 | needed. The number of threads are chosen based on the number of the CPU | |
115 | cores of your system although you can change it via -j (--jobs) option. | |
116 | ||
117 | ||
118 | Toolchains | |
119 | ---------- | |
120 | ||
121 | Appropriate toolchain are necessary to generate include/autoconf.mk | |
122 | for all the architectures supported by U-Boot. Most of them are available | |
123 | at the kernel.org site, some are not provided by kernel.org. | |
124 | ||
125 | The default per-arch CROSS_COMPILE used by this tool is specified by | |
126 | the list below, CROSS_COMPILE. You may wish to update the list to | |
127 | use your own. Instead of modifying the list directly, you can give | |
128 | them via environments. | |
129 | ||
130 | ||
131 | Available options | |
132 | ----------------- | |
133 | ||
134 | -c, --color | |
135 | Surround each portion of the log with escape sequences to display it | |
136 | in color on the terminal. | |
137 | ||
91040e85 JH |
138 | -d, --defconfigs |
139 | Specify a file containing a list of defconfigs to move | |
140 | ||
5a27c734 MY |
141 | -n, --dry-run |
142 | Peform a trial run that does not make any changes. It is useful to | |
143 | see what is going to happen before one actually runs it. | |
144 | ||
145 | -e, --exit-on-error | |
146 | Exit immediately if Make exits with a non-zero status while processing | |
147 | a defconfig file. | |
148 | ||
2144f880 JH |
149 | -H, --headers-only |
150 | Only cleanup the headers; skip the defconfig processing | |
151 | ||
5a27c734 MY |
152 | -j, --jobs |
153 | Specify the number of threads to run simultaneously. If not specified, | |
154 | the number of threads is the same as the number of CPU cores. | |
155 | ||
95bf9c7e JH |
156 | -v, --verbose |
157 | Show any build errors as boards are built | |
158 | ||
5a27c734 MY |
159 | To see the complete list of supported options, run |
160 | ||
161 | $ tools/moveconfig.py -h | |
162 | ||
163 | """ | |
164 | ||
165 | import fnmatch | |
166 | import multiprocessing | |
167 | import optparse | |
168 | import os | |
169 | import re | |
170 | import shutil | |
171 | import subprocess | |
172 | import sys | |
173 | import tempfile | |
174 | import time | |
175 | ||
176 | SHOW_GNU_MAKE = 'scripts/show-gnu-make' | |
177 | SLEEP_TIME=0.03 | |
178 | ||
179 | # Here is the list of cross-tools I use. | |
180 | # Most of them are available at kernel.org | |
181 | # (https://www.kernel.org/pub/tools/crosstool/files/bin/), except the followings: | |
182 | # arc: https://github.com/foss-for-synopsys-dwc-arc-processors/toolchain/releases | |
183 | # blackfin: http://sourceforge.net/projects/adi-toolchain/files/ | |
4440ecec | 184 | # nds32: http://osdk.andestech.com/packages/nds32le-linux-glibc-v1.tgz |
5a27c734 MY |
185 | # nios2: https://sourcery.mentor.com/GNUToolchain/subscription42545 |
186 | # sh: http://sourcery.mentor.com/public/gnu_toolchain/sh-linux-gnu | |
e8aebc47 BM |
187 | # |
188 | # openrisc kernel.org toolchain is out of date, download latest one from | |
189 | # http://opencores.org/or1k/OpenRISC_GNU_tool_chain#Prebuilt_versions | |
5a27c734 MY |
190 | CROSS_COMPILE = { |
191 | 'arc': 'arc-linux-', | |
192 | 'aarch64': 'aarch64-linux-', | |
193 | 'arm': 'arm-unknown-linux-gnueabi-', | |
194 | 'avr32': 'avr32-linux-', | |
195 | 'blackfin': 'bfin-elf-', | |
196 | 'm68k': 'm68k-linux-', | |
197 | 'microblaze': 'microblaze-linux-', | |
198 | 'mips': 'mips-linux-', | |
199 | 'nds32': 'nds32le-linux-', | |
200 | 'nios2': 'nios2-linux-gnu-', | |
e8aebc47 | 201 | 'openrisc': 'or1k-elf-', |
5a27c734 MY |
202 | 'powerpc': 'powerpc-linux-', |
203 | 'sh': 'sh-linux-gnu-', | |
204 | 'sparc': 'sparc-linux-', | |
205 | 'x86': 'i386-linux-' | |
206 | } | |
207 | ||
208 | STATE_IDLE = 0 | |
209 | STATE_DEFCONFIG = 1 | |
210 | STATE_AUTOCONF = 2 | |
96464bad | 211 | STATE_SAVEDEFCONFIG = 3 |
5a27c734 MY |
212 | |
213 | ACTION_MOVE = 0 | |
214 | ACTION_DEFAULT_VALUE = 1 | |
215 | ACTION_ALREADY_EXIST = 2 | |
216 | ACTION_UNDEFINED = 3 | |
217 | ||
218 | COLOR_BLACK = '0;30' | |
219 | COLOR_RED = '0;31' | |
220 | COLOR_GREEN = '0;32' | |
221 | COLOR_BROWN = '0;33' | |
222 | COLOR_BLUE = '0;34' | |
223 | COLOR_PURPLE = '0;35' | |
224 | COLOR_CYAN = '0;36' | |
225 | COLOR_LIGHT_GRAY = '0;37' | |
226 | COLOR_DARK_GRAY = '1;30' | |
227 | COLOR_LIGHT_RED = '1;31' | |
228 | COLOR_LIGHT_GREEN = '1;32' | |
229 | COLOR_YELLOW = '1;33' | |
230 | COLOR_LIGHT_BLUE = '1;34' | |
231 | COLOR_LIGHT_PURPLE = '1;35' | |
232 | COLOR_LIGHT_CYAN = '1;36' | |
233 | COLOR_WHITE = '1;37' | |
234 | ||
235 | ### helper functions ### | |
236 | def get_devnull(): | |
237 | """Get the file object of '/dev/null' device.""" | |
238 | try: | |
239 | devnull = subprocess.DEVNULL # py3k | |
240 | except AttributeError: | |
241 | devnull = open(os.devnull, 'wb') | |
242 | return devnull | |
243 | ||
244 | def check_top_directory(): | |
245 | """Exit if we are not at the top of source directory.""" | |
246 | for f in ('README', 'Licenses'): | |
247 | if not os.path.exists(f): | |
248 | sys.exit('Please run at the top of source directory.') | |
249 | ||
250 | def get_make_cmd(): | |
251 | """Get the command name of GNU Make. | |
252 | ||
253 | U-Boot needs GNU Make for building, but the command name is not | |
254 | necessarily "make". (for example, "gmake" on FreeBSD). | |
255 | Returns the most appropriate command name on your system. | |
256 | """ | |
257 | process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) | |
258 | ret = process.communicate() | |
259 | if process.returncode: | |
260 | sys.exit('GNU Make not found') | |
261 | return ret[0].rstrip() | |
262 | ||
263 | def color_text(color_enabled, color, string): | |
264 | """Return colored string.""" | |
265 | if color_enabled: | |
266 | return '\033[' + color + 'm' + string + '\033[0m' | |
267 | else: | |
268 | return string | |
269 | ||
270 | def log_msg(color_enabled, color, defconfig, msg): | |
271 | """Return the formated line for the log.""" | |
272 | return defconfig[:-len('_defconfig')].ljust(37) + ': ' + \ | |
273 | color_text(color_enabled, color, msg) + '\n' | |
274 | ||
275 | def update_cross_compile(): | |
1cc0a9f4 | 276 | """Update per-arch CROSS_COMPILE via environment variables |
5a27c734 MY |
277 | |
278 | The default CROSS_COMPILE values are available | |
279 | in the CROSS_COMPILE list above. | |
280 | ||
1cc0a9f4 | 281 | You can override them via environment variables |
5a27c734 MY |
282 | CROSS_COMPILE_{ARCH}. |
283 | ||
284 | For example, if you want to override toolchain prefixes | |
285 | for ARM and PowerPC, you can do as follows in your shell: | |
286 | ||
287 | export CROSS_COMPILE_ARM=... | |
288 | export CROSS_COMPILE_POWERPC=... | |
289 | """ | |
290 | archs = [] | |
291 | ||
292 | for arch in os.listdir('arch'): | |
293 | if os.path.exists(os.path.join('arch', arch, 'Makefile')): | |
294 | archs.append(arch) | |
295 | ||
296 | # arm64 is a special case | |
297 | archs.append('aarch64') | |
298 | ||
299 | for arch in archs: | |
300 | env = 'CROSS_COMPILE_' + arch.upper() | |
301 | cross_compile = os.environ.get(env) | |
302 | if cross_compile: | |
303 | CROSS_COMPILE[arch] = cross_compile | |
304 | ||
305 | def cleanup_one_header(header_path, patterns, dry_run): | |
306 | """Clean regex-matched lines away from a file. | |
307 | ||
308 | Arguments: | |
309 | header_path: path to the cleaned file. | |
310 | patterns: list of regex patterns. Any lines matching to these | |
311 | patterns are deleted. | |
312 | dry_run: make no changes, but still display log. | |
313 | """ | |
314 | with open(header_path) as f: | |
315 | lines = f.readlines() | |
316 | ||
317 | matched = [] | |
318 | for i, line in enumerate(lines): | |
319 | for pattern in patterns: | |
320 | m = pattern.search(line) | |
321 | if m: | |
322 | print '%s: %s: %s' % (header_path, i + 1, line), | |
323 | matched.append(i) | |
324 | break | |
325 | ||
326 | if dry_run or not matched: | |
327 | return | |
328 | ||
329 | with open(header_path, 'w') as f: | |
330 | for i, line in enumerate(lines): | |
331 | if not i in matched: | |
332 | f.write(line) | |
333 | ||
334 | def cleanup_headers(config_attrs, dry_run): | |
335 | """Delete config defines from board headers. | |
336 | ||
337 | Arguments: | |
338 | config_attrs: A list of dictionaris, each of them includes the name, | |
339 | the type, and the default value of the target config. | |
340 | dry_run: make no changes, but still display log. | |
341 | """ | |
342 | while True: | |
343 | choice = raw_input('Clean up headers? [y/n]: ').lower() | |
344 | print choice | |
345 | if choice == 'y' or choice == 'n': | |
346 | break | |
347 | ||
348 | if choice == 'n': | |
349 | return | |
350 | ||
351 | patterns = [] | |
352 | for config_attr in config_attrs: | |
353 | config = config_attr['config'] | |
354 | patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) | |
355 | patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) | |
356 | ||
60727f51 JH |
357 | for dir in 'include', 'arch', 'board': |
358 | for (dirpath, dirnames, filenames) in os.walk(dir): | |
359 | for filename in filenames: | |
360 | if not fnmatch.fnmatch(filename, '*~'): | |
361 | cleanup_one_header(os.path.join(dirpath, filename), | |
362 | patterns, dry_run) | |
5a27c734 MY |
363 | |
364 | ### classes ### | |
365 | class KconfigParser: | |
366 | ||
367 | """A parser of .config and include/autoconf.mk.""" | |
368 | ||
369 | re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') | |
370 | re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') | |
371 | ||
372 | def __init__(self, config_attrs, options, build_dir): | |
373 | """Create a new parser. | |
374 | ||
375 | Arguments: | |
376 | config_attrs: A list of dictionaris, each of them includes the name, | |
377 | the type, and the default value of the target config. | |
378 | options: option flags. | |
379 | build_dir: Build directory. | |
380 | """ | |
381 | self.config_attrs = config_attrs | |
382 | self.options = options | |
383 | self.build_dir = build_dir | |
384 | ||
385 | def get_cross_compile(self): | |
386 | """Parse .config file and return CROSS_COMPILE. | |
387 | ||
388 | Returns: | |
389 | A string storing the compiler prefix for the architecture. | |
390 | """ | |
391 | arch = '' | |
392 | cpu = '' | |
393 | dotconfig = os.path.join(self.build_dir, '.config') | |
394 | for line in open(dotconfig): | |
395 | m = self.re_arch.match(line) | |
396 | if m: | |
397 | arch = m.group(1) | |
398 | continue | |
399 | m = self.re_cpu.match(line) | |
400 | if m: | |
401 | cpu = m.group(1) | |
402 | ||
403 | assert arch, 'Error: arch is not defined in %s' % defconfig | |
404 | ||
405 | # fix-up for aarch64 | |
406 | if arch == 'arm' and cpu == 'armv8': | |
407 | arch = 'aarch64' | |
408 | ||
409 | return CROSS_COMPILE.get(arch, '') | |
410 | ||
96464bad | 411 | def parse_one_config(self, config_attr, defconfig_lines, autoconf_lines): |
5a27c734 MY |
412 | """Parse .config, defconfig, include/autoconf.mk for one config. |
413 | ||
414 | This function looks for the config options in the lines from | |
415 | defconfig, .config, and include/autoconf.mk in order to decide | |
416 | which action should be taken for this defconfig. | |
417 | ||
418 | Arguments: | |
419 | config_attr: A dictionary including the name, the type, | |
420 | and the default value of the target config. | |
421 | defconfig_lines: lines from the original defconfig file. | |
5a27c734 MY |
422 | autoconf_lines: lines from the include/autoconf.mk file. |
423 | ||
424 | Returns: | |
425 | A tupple of the action for this defconfig and the line | |
426 | matched for the config. | |
427 | """ | |
428 | config = config_attr['config'] | |
429 | not_set = '# %s is not set' % config | |
430 | ||
431 | if config_attr['type'] in ('bool', 'tristate') and \ | |
432 | config_attr['default'] == 'n': | |
433 | default = not_set | |
434 | else: | |
435 | default = config + '=' + config_attr['default'] | |
436 | ||
96464bad | 437 | for line in defconfig_lines: |
5a27c734 MY |
438 | line = line.rstrip() |
439 | if line.startswith(config + '=') or line == not_set: | |
440 | return (ACTION_ALREADY_EXIST, line) | |
441 | ||
442 | if config_attr['type'] in ('bool', 'tristate'): | |
443 | value = not_set | |
444 | else: | |
445 | value = '(undefined)' | |
446 | ||
447 | for line in autoconf_lines: | |
448 | line = line.rstrip() | |
449 | if line.startswith(config + '='): | |
450 | value = line | |
451 | break | |
452 | ||
453 | if value == default: | |
454 | action = ACTION_DEFAULT_VALUE | |
455 | elif value == '(undefined)': | |
456 | action = ACTION_UNDEFINED | |
457 | else: | |
458 | action = ACTION_MOVE | |
459 | ||
460 | return (action, value) | |
461 | ||
462 | def update_defconfig(self, defconfig): | |
463 | """Parse files for the config options and update the defconfig. | |
464 | ||
465 | This function parses the given defconfig, the generated .config | |
466 | and include/autoconf.mk searching the target options. | |
467 | Move the config option(s) to the defconfig or do nothing if unneeded. | |
468 | Also, display the log to show what happened to this defconfig. | |
469 | ||
470 | Arguments: | |
471 | defconfig: defconfig name. | |
472 | """ | |
473 | ||
474 | defconfig_path = os.path.join('configs', defconfig) | |
475 | dotconfig_path = os.path.join(self.build_dir, '.config') | |
476 | autoconf_path = os.path.join(self.build_dir, 'include', 'autoconf.mk') | |
477 | results = [] | |
478 | ||
479 | with open(defconfig_path) as f: | |
480 | defconfig_lines = f.readlines() | |
481 | ||
5a27c734 MY |
482 | with open(autoconf_path) as f: |
483 | autoconf_lines = f.readlines() | |
484 | ||
485 | for config_attr in self.config_attrs: | |
486 | result = self.parse_one_config(config_attr, defconfig_lines, | |
96464bad | 487 | autoconf_lines) |
5a27c734 MY |
488 | results.append(result) |
489 | ||
490 | log = '' | |
491 | ||
492 | for (action, value) in results: | |
493 | if action == ACTION_MOVE: | |
494 | actlog = "Move '%s'" % value | |
495 | log_color = COLOR_LIGHT_GREEN | |
496 | elif action == ACTION_DEFAULT_VALUE: | |
497 | actlog = "Default value '%s'. Do nothing." % value | |
498 | log_color = COLOR_LIGHT_BLUE | |
499 | elif action == ACTION_ALREADY_EXIST: | |
500 | actlog = "'%s' already defined in Kconfig. Do nothing." % value | |
501 | log_color = COLOR_LIGHT_PURPLE | |
502 | elif action == ACTION_UNDEFINED: | |
503 | actlog = "Undefined. Do nothing." | |
504 | log_color = COLOR_DARK_GRAY | |
505 | else: | |
506 | sys.exit("Internal Error. This should not happen.") | |
507 | ||
508 | log += log_msg(self.options.color, log_color, defconfig, actlog) | |
509 | ||
510 | # Some threads are running in parallel. | |
511 | # Print log in one shot to not mix up logs from different threads. | |
512 | print log, | |
513 | ||
514 | if not self.options.dry_run: | |
96464bad | 515 | with open(dotconfig_path, 'a') as f: |
5a27c734 MY |
516 | for (action, value) in results: |
517 | if action == ACTION_MOVE: | |
518 | f.write(value + '\n') | |
519 | ||
520 | os.remove(os.path.join(self.build_dir, 'include', 'config', 'auto.conf')) | |
521 | os.remove(autoconf_path) | |
522 | ||
523 | class Slot: | |
524 | ||
525 | """A slot to store a subprocess. | |
526 | ||
527 | Each instance of this class handles one subprocess. | |
528 | This class is useful to control multiple threads | |
529 | for faster processing. | |
530 | """ | |
531 | ||
532 | def __init__(self, config_attrs, options, devnull, make_cmd): | |
533 | """Create a new process slot. | |
534 | ||
535 | Arguments: | |
536 | config_attrs: A list of dictionaris, each of them includes the name, | |
537 | the type, and the default value of the target config. | |
538 | options: option flags. | |
539 | devnull: A file object of '/dev/null'. | |
540 | make_cmd: command name of GNU Make. | |
541 | """ | |
542 | self.options = options | |
543 | self.build_dir = tempfile.mkdtemp() | |
544 | self.devnull = devnull | |
545 | self.make_cmd = (make_cmd, 'O=' + self.build_dir) | |
546 | self.parser = KconfigParser(config_attrs, options, self.build_dir) | |
547 | self.state = STATE_IDLE | |
548 | self.failed_boards = [] | |
549 | ||
550 | def __del__(self): | |
551 | """Delete the working directory | |
552 | ||
553 | This function makes sure the temporary directory is cleaned away | |
554 | even if Python suddenly dies due to error. It should be done in here | |
555 | because it is guranteed the destructor is always invoked when the | |
556 | instance of the class gets unreferenced. | |
557 | ||
558 | If the subprocess is still running, wait until it finishes. | |
559 | """ | |
560 | if self.state != STATE_IDLE: | |
561 | while self.ps.poll() == None: | |
562 | pass | |
563 | shutil.rmtree(self.build_dir) | |
564 | ||
2e2ce6c0 | 565 | def add(self, defconfig, num, total): |
5a27c734 MY |
566 | """Assign a new subprocess for defconfig and add it to the slot. |
567 | ||
568 | If the slot is vacant, create a new subprocess for processing the | |
569 | given defconfig and add it to the slot. Just returns False if | |
570 | the slot is occupied (i.e. the current subprocess is still running). | |
571 | ||
572 | Arguments: | |
573 | defconfig: defconfig name. | |
574 | ||
575 | Returns: | |
576 | Return True on success or False on failure | |
577 | """ | |
578 | if self.state != STATE_IDLE: | |
579 | return False | |
580 | cmd = list(self.make_cmd) | |
581 | cmd.append(defconfig) | |
25400090 JH |
582 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, |
583 | stderr=subprocess.PIPE) | |
5a27c734 MY |
584 | self.defconfig = defconfig |
585 | self.state = STATE_DEFCONFIG | |
2e2ce6c0 JH |
586 | self.num = num |
587 | self.total = total | |
5a27c734 MY |
588 | return True |
589 | ||
590 | def poll(self): | |
591 | """Check the status of the subprocess and handle it as needed. | |
592 | ||
593 | Returns True if the slot is vacant (i.e. in idle state). | |
594 | If the configuration is successfully finished, assign a new | |
595 | subprocess to build include/autoconf.mk. | |
596 | If include/autoconf.mk is generated, invoke the parser to | |
597 | parse the .config and the include/autoconf.mk, and then set the | |
598 | slot back to the idle state. | |
599 | ||
600 | Returns: | |
601 | Return True if the subprocess is terminated, False otherwise | |
602 | """ | |
603 | if self.state == STATE_IDLE: | |
604 | return True | |
605 | ||
606 | if self.ps.poll() == None: | |
607 | return False | |
608 | ||
609 | if self.ps.poll() != 0: | |
25400090 JH |
610 | errmsg = 'Failed to process.' |
611 | errout = self.ps.stderr.read() | |
612 | if errout.find('gcc: command not found') != -1: | |
613 | errmsg = 'Compiler not found (' | |
614 | errmsg += color_text(self.options.color, COLOR_YELLOW, | |
615 | self.cross_compile) | |
616 | errmsg += color_text(self.options.color, COLOR_LIGHT_RED, | |
617 | ')') | |
5a27c734 MY |
618 | print >> sys.stderr, log_msg(self.options.color, |
619 | COLOR_LIGHT_RED, | |
620 | self.defconfig, | |
25400090 | 621 | errmsg), |
95bf9c7e JH |
622 | if self.options.verbose: |
623 | print >> sys.stderr, color_text(self.options.color, | |
624 | COLOR_LIGHT_CYAN, errout) | |
5a27c734 MY |
625 | if self.options.exit_on_error: |
626 | sys.exit("Exit on error.") | |
627 | else: | |
628 | # If --exit-on-error flag is not set, | |
629 | # skip this board and continue. | |
630 | # Record the failed board. | |
631 | self.failed_boards.append(self.defconfig) | |
632 | self.state = STATE_IDLE | |
633 | return True | |
634 | ||
635 | if self.state == STATE_AUTOCONF: | |
636 | self.parser.update_defconfig(self.defconfig) | |
96464bad | 637 | |
2e2ce6c0 JH |
638 | print ' %d defconfigs out of %d\r' % (self.num + 1, self.total), |
639 | sys.stdout.flush() | |
640 | ||
96464bad JH |
641 | """Save off the defconfig in a consistent way""" |
642 | cmd = list(self.make_cmd) | |
643 | cmd.append('savedefconfig') | |
644 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
25400090 | 645 | stderr=subprocess.PIPE) |
96464bad JH |
646 | self.state = STATE_SAVEDEFCONFIG |
647 | return False | |
648 | ||
649 | if self.state == STATE_SAVEDEFCONFIG: | |
650 | defconfig_path = os.path.join(self.build_dir, 'defconfig') | |
651 | shutil.move(defconfig_path, | |
652 | os.path.join('configs', self.defconfig)) | |
5a27c734 MY |
653 | self.state = STATE_IDLE |
654 | return True | |
655 | ||
25400090 | 656 | self.cross_compile = self.parser.get_cross_compile() |
5a27c734 | 657 | cmd = list(self.make_cmd) |
25400090 JH |
658 | if self.cross_compile: |
659 | cmd.append('CROSS_COMPILE=%s' % self.cross_compile) | |
7740f653 | 660 | cmd.append('KCONFIG_IGNORE_DUPLICATES=1') |
5a27c734 | 661 | cmd.append('include/config/auto.conf') |
25400090 JH |
662 | """This will be screen-scraped, so be sure the expected text will be |
663 | returned consistently on every machine by setting LANG=C""" | |
664 | self.ps = subprocess.Popen(cmd, stdout=self.devnull, | |
665 | env=dict(os.environ, LANG='C'), | |
666 | stderr=subprocess.PIPE) | |
5a27c734 MY |
667 | self.state = STATE_AUTOCONF |
668 | return False | |
669 | ||
670 | def get_failed_boards(self): | |
671 | """Returns a list of failed boards (defconfigs) in this slot. | |
672 | """ | |
673 | return self.failed_boards | |
674 | ||
675 | class Slots: | |
676 | ||
677 | """Controller of the array of subprocess slots.""" | |
678 | ||
679 | def __init__(self, config_attrs, options): | |
680 | """Create a new slots controller. | |
681 | ||
682 | Arguments: | |
683 | config_attrs: A list of dictionaris containing the name, the type, | |
684 | and the default value of the target CONFIG. | |
685 | options: option flags. | |
686 | """ | |
687 | self.options = options | |
688 | self.slots = [] | |
689 | devnull = get_devnull() | |
690 | make_cmd = get_make_cmd() | |
691 | for i in range(options.jobs): | |
692 | self.slots.append(Slot(config_attrs, options, devnull, make_cmd)) | |
693 | ||
2e2ce6c0 | 694 | def add(self, defconfig, num, total): |
5a27c734 MY |
695 | """Add a new subprocess if a vacant slot is found. |
696 | ||
697 | Arguments: | |
698 | defconfig: defconfig name to be put into. | |
699 | ||
700 | Returns: | |
701 | Return True on success or False on failure | |
702 | """ | |
703 | for slot in self.slots: | |
2e2ce6c0 | 704 | if slot.add(defconfig, num, total): |
5a27c734 MY |
705 | return True |
706 | return False | |
707 | ||
708 | def available(self): | |
709 | """Check if there is a vacant slot. | |
710 | ||
711 | Returns: | |
712 | Return True if at lease one vacant slot is found, False otherwise. | |
713 | """ | |
714 | for slot in self.slots: | |
715 | if slot.poll(): | |
716 | return True | |
717 | return False | |
718 | ||
719 | def empty(self): | |
720 | """Check if all slots are vacant. | |
721 | ||
722 | Returns: | |
723 | Return True if all the slots are vacant, False otherwise. | |
724 | """ | |
725 | ret = True | |
726 | for slot in self.slots: | |
727 | if not slot.poll(): | |
728 | ret = False | |
729 | return ret | |
730 | ||
731 | def show_failed_boards(self): | |
732 | """Display all of the failed boards (defconfigs).""" | |
733 | failed_boards = [] | |
734 | ||
735 | for slot in self.slots: | |
736 | failed_boards += slot.get_failed_boards() | |
737 | ||
738 | if len(failed_boards) > 0: | |
739 | msg = [ "The following boards were not processed due to error:" ] | |
740 | msg += failed_boards | |
741 | for line in msg: | |
742 | print >> sys.stderr, color_text(self.options.color, | |
743 | COLOR_LIGHT_RED, line) | |
744 | ||
2559cd89 JH |
745 | with open('moveconfig.failed', 'w') as f: |
746 | for board in failed_boards: | |
747 | f.write(board + '\n') | |
748 | ||
5a27c734 MY |
749 | def move_config(config_attrs, options): |
750 | """Move config options to defconfig files. | |
751 | ||
752 | Arguments: | |
753 | config_attrs: A list of dictionaris, each of them includes the name, | |
754 | the type, and the default value of the target config. | |
755 | options: option flags | |
756 | """ | |
5a27c734 MY |
757 | if len(config_attrs) == 0: |
758 | print 'Nothing to do. exit.' | |
759 | sys.exit(0) | |
760 | ||
761 | print 'Move the following CONFIG options (jobs: %d)' % options.jobs | |
762 | for config_attr in config_attrs: | |
763 | print ' %s (type: %s, default: %s)' % (config_attr['config'], | |
764 | config_attr['type'], | |
765 | config_attr['default']) | |
766 | ||
91040e85 JH |
767 | if options.defconfigs: |
768 | defconfigs = [line.strip() for line in open(options.defconfigs)] | |
769 | for i, defconfig in enumerate(defconfigs): | |
770 | if not defconfig.endswith('_defconfig'): | |
771 | defconfigs[i] = defconfig + '_defconfig' | |
772 | if not os.path.exists(os.path.join('configs', defconfigs[i])): | |
773 | sys.exit('%s - defconfig does not exist. Stopping.' % | |
774 | defconfigs[i]) | |
775 | else: | |
776 | # All the defconfig files to be processed | |
777 | defconfigs = [] | |
778 | for (dirpath, dirnames, filenames) in os.walk('configs'): | |
779 | dirpath = dirpath[len('configs') + 1:] | |
780 | for filename in fnmatch.filter(filenames, '*_defconfig'): | |
781 | defconfigs.append(os.path.join(dirpath, filename)) | |
5a27c734 MY |
782 | |
783 | slots = Slots(config_attrs, options) | |
784 | ||
785 | # Main loop to process defconfig files: | |
786 | # Add a new subprocess into a vacant slot. | |
787 | # Sleep if there is no available slot. | |
2e2ce6c0 JH |
788 | for i, defconfig in enumerate(defconfigs): |
789 | while not slots.add(defconfig, i, len(defconfigs)): | |
5a27c734 MY |
790 | while not slots.available(): |
791 | # No available slot: sleep for a while | |
792 | time.sleep(SLEEP_TIME) | |
793 | ||
794 | # wait until all the subprocesses finish | |
795 | while not slots.empty(): | |
796 | time.sleep(SLEEP_TIME) | |
797 | ||
2e2ce6c0 | 798 | print '' |
5a27c734 MY |
799 | slots.show_failed_boards() |
800 | ||
5a27c734 MY |
801 | def bad_recipe(filename, linenum, msg): |
802 | """Print error message with the file name and the line number and exit.""" | |
803 | sys.exit("%s: line %d: error : " % (filename, linenum) + msg) | |
804 | ||
805 | def parse_recipe(filename): | |
806 | """Parse the recipe file and retrieve the config attributes. | |
807 | ||
808 | This function parses the given recipe file and gets the name, | |
809 | the type, and the default value of the target config options. | |
810 | ||
811 | Arguments: | |
812 | filename: path to file to be parsed. | |
813 | Returns: | |
814 | A list of dictionaris, each of them includes the name, | |
815 | the type, and the default value of the target config. | |
816 | """ | |
817 | config_attrs = [] | |
818 | linenum = 1 | |
819 | ||
820 | for line in open(filename): | |
821 | tokens = line.split() | |
822 | if len(tokens) != 3: | |
823 | bad_recipe(filename, linenum, | |
824 | "%d fields in this line. Each line must contain 3 fields" | |
825 | % len(tokens)) | |
826 | ||
827 | (config, type, default) = tokens | |
828 | ||
829 | # prefix the option name with CONFIG_ if missing | |
830 | if not config.startswith('CONFIG_'): | |
831 | config = 'CONFIG_' + config | |
832 | ||
833 | # sanity check of default values | |
834 | if type == 'bool': | |
835 | if not default in ('y', 'n'): | |
836 | bad_recipe(filename, linenum, | |
837 | "default for bool type must be either y or n") | |
838 | elif type == 'tristate': | |
839 | if not default in ('y', 'm', 'n'): | |
840 | bad_recipe(filename, linenum, | |
841 | "default for tristate type must be y, m, or n") | |
842 | elif type == 'string': | |
843 | if default[0] != '"' or default[-1] != '"': | |
844 | bad_recipe(filename, linenum, | |
845 | "default for string type must be surrounded by double-quotations") | |
846 | elif type == 'int': | |
847 | try: | |
848 | int(default) | |
849 | except: | |
850 | bad_recipe(filename, linenum, | |
851 | "type is int, but default value is not decimal") | |
852 | elif type == 'hex': | |
853 | if len(default) < 2 or default[:2] != '0x': | |
854 | bad_recipe(filename, linenum, | |
855 | "default for hex type must be prefixed with 0x") | |
856 | try: | |
857 | int(default, 16) | |
858 | except: | |
859 | bad_recipe(filename, linenum, | |
860 | "type is hex, but default value is not hexadecimal") | |
861 | else: | |
862 | bad_recipe(filename, linenum, | |
863 | "unsupported type '%s'. type must be one of bool, tristate, string, int, hex" | |
864 | % type) | |
865 | ||
866 | config_attrs.append({'config': config, 'type': type, 'default': default}) | |
867 | linenum += 1 | |
868 | ||
869 | return config_attrs | |
870 | ||
871 | def main(): | |
872 | try: | |
873 | cpu_count = multiprocessing.cpu_count() | |
874 | except NotImplementedError: | |
875 | cpu_count = 1 | |
876 | ||
877 | parser = optparse.OptionParser() | |
878 | # Add options here | |
879 | parser.add_option('-c', '--color', action='store_true', default=False, | |
880 | help='display the log in color') | |
91040e85 JH |
881 | parser.add_option('-d', '--defconfigs', type='string', |
882 | help='a file containing a list of defconfigs to move') | |
5a27c734 MY |
883 | parser.add_option('-n', '--dry-run', action='store_true', default=False, |
884 | help='perform a trial run (show log with no changes)') | |
885 | parser.add_option('-e', '--exit-on-error', action='store_true', | |
886 | default=False, | |
887 | help='exit immediately on any error') | |
2144f880 JH |
888 | parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', |
889 | action='store_true', default=False, | |
890 | help='only cleanup the headers') | |
5a27c734 MY |
891 | parser.add_option('-j', '--jobs', type='int', default=cpu_count, |
892 | help='the number of jobs to run simultaneously') | |
95bf9c7e JH |
893 | parser.add_option('-v', '--verbose', action='store_true', default=False, |
894 | help='show any build errors as boards are built') | |
5a27c734 MY |
895 | parser.usage += ' recipe_file\n\n' + \ |
896 | 'The recipe_file should describe config options you want to move.\n' + \ | |
897 | 'Each line should contain config_name, type, default_value\n\n' + \ | |
898 | 'Example:\n' + \ | |
899 | 'CONFIG_FOO bool n\n' + \ | |
900 | 'CONFIG_BAR int 100\n' + \ | |
901 | 'CONFIG_BAZ string "hello"\n' | |
902 | ||
903 | (options, args) = parser.parse_args() | |
904 | ||
905 | if len(args) != 1: | |
906 | parser.print_usage() | |
907 | sys.exit(1) | |
908 | ||
909 | config_attrs = parse_recipe(args[0]) | |
910 | ||
911 | update_cross_compile() | |
912 | ||
2144f880 JH |
913 | check_top_directory() |
914 | ||
915 | if not options.cleanup_headers_only: | |
916 | move_config(config_attrs, options) | |
917 | ||
918 | cleanup_headers(config_attrs, options.dry_run) | |
5a27c734 MY |
919 | |
920 | if __name__ == '__main__': | |
921 | main() |