-
Notifications
You must be signed in to change notification settings - Fork 28
/
CHANGES
2332 lines (1339 loc) · 73.5 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
3.0.1-43 | 2024-12-10 12:21:04 -0700
* Update versions for pre-commit hooks (Tim Wojtulewicz, Corelight)
3.0.1-41 | 2024-09-10 10:36:13 +0200
* Fix UnicodeEncodeError in `zkg create` with non-ASCII characters in user vars (Fupeng Zhao)
3.0.1-39 | 2024-07-18 18:48:44 +0200
* Use documented Git package exception API (Benjamin Bannier, Corelight)
* Bump pre-commit hooks (Benjamin Bannier, Corelight)
* Use f-strings in more places (Benjamin Bannier, Corelight)
* Add installation workaround for tar files with macos metadata (Benjamin Bannier, Corelight)
* Fix test use of `sed` for macos (Benjamin Bannier, Corelight)
This fixes the invocation of `sed` in the test to be compatible with
macos' `sed` version. Previously this test would fail with
```
sed: 1: "./install-dir/bin/zkg": invalid command code
```
3.0.1-33 | 2024-05-13 13:52:59 -0700
* Pull in `btest` when installing package manager (Benjamin Bannier, Corelight)
This reverts a24124d2f298b306e47d43a8aa0361f98ede9a04 where we moved
`btest` into a dev dependency which would not be installed by default.
While this would have been unobservable for users getting their `zkg`
via Zeek (which also provides `btest`), or who install `zkg` via `pipx`
(which even previously would never have linked the pulled in `btest`
into the prefix), it might break users installing via `pip` into a
random prefix.
3.0.1-31 | 2024-05-08 09:57:29 -0700
* Make requirements.txt match pyproject file (Tim Wojtulewicz, Corelight)
Notably, the two versions listed here are the current releases. The current
setting causes RTD to fail since it can't find versions newer than those.
3.0.1-29 | 2024-05-08 09:38:06 -0700
* Remove python2 `urlparse` (Benjamin Bannier)
* Require at least python-3.9 (Benjamin Bannier)
* Enforce pyflakes lint set (Benjamin Bannier)
* Replace `str.format` uses with f-strings (Benjamin Bannier)
* Enforce flake8-bugbear rules (Benjamin Bannier)
* Enforce trainling commas (Benjamin Bannier)
* Remove shadowing of builtins (Benjamin Bannier)
* Remove unnecessary comprehensions (Benjamin Bannier)
* Fix accidental implicit str concats (Benjamin Bannier)
* Enforce PEP8 naming conventions (Benjamin Bannier)
* Enable RUF lint set (Benjamin Bannier)
* Fix imports with ruff (Benjamin Bannier)
* Modernize Python code with ruff (Benjamin Bannier)
* Move btest to dev dependencies (Benjamin Bannier)
* Modernize doc module with pyupgrade (Benjamin Bannier)
* Automatically reformat doc module (Benjamin Bannier)
* Fix ruff lints in doc modules (Benjamin Bannier)
* Stop excluding `doc/ext/sphinxarg` from linting (Benjamin Bannier)
* Bump pre-commit hooks (Benjamin Bannier)
* Centralize project configuration (Benjamin Bannier)
We previously used a mix of `setup.py` and `setup.cfg` which are not
well supported by more modern Python developer tools which instead
prefer `pyproject.toml`. This meant that developing this project
required manually setting up and managing virtualenvs.
With this patch we now move most of our configuration over into
`pyproject.toml` so especially dev environments can be managed with
tools like Hatch[^1] or Rye[^2]. Dev envs managed by these tools are
automaticallt discovered by many editors so there is potentially less
need to even think about virtualenvs.
Now `setup.py` contains the absolute minimum required settings. We still
need to keep dynamically generating the package version from the
Zeek-style `VERSION` file, and also need to add a tweak to make sure it
is distributed in wheels so the installation code reading it has access
to it.
Since settings in `setup.cfg` can conflict with `pyproject.toml` we
delete the file after either moving all setttings to `pyproject.toml` or
to tool specific files (flake8 still does not support `pyproject.toml`).
For now we keep `requirements.txt` (which updated versions) so we can
install dev dependencies, e.g., for RTD. Unfortunately dev dependencies
in `pyproject.toml` are still not standardized[^3].
[^1]: hatch.pypa.io
[^2]: https://rye-up.com/
[^3]: https://discuss.python.org/t/development-dependencies-in-pyproject-toml/26149
3.0.1-8 | 2024-04-10 09:34:51 -0700
* Standardize on ruff for python formatting/linting via pre-commits (Tim Wojtulewicz, Corelight)
3.0.1-6 | 2024-03-12 11:09:28 -0700
* Use default Python version for GH pre-commit workflow (Benjamin Bannier, Corelight)
I am unsure why we pinned this to the concrete version in the initial
commit. Looking at it now pinning a linting workflow to e.g., our
minimal Python version seems to make no sense as it does not actually
run any code. Additionally it introduces a requirement of all pre-commit
hooks working with our lowest possible version; this is not the case for
e.g., `pyupgrade` which requires python-3.8 now.
* Bump pyupgrade pre-commit to latest version (Tim Wojtulewicz, Corelight)
3.0.1-3 | 2024-03-01 15:48:40 -0700
* Extend recognition of local Git repos (Benjamin Bannier, Corelight)
We previously would only recognize local Git repos for validation only
if their path started with `./` or `/`, even tough installation would
still find them without these prefixes.
Closes #181.
* Add `zkg.meta` to test package (Benjamin Bannier, Corelight)
3.0.1 | 2024-02-20 10:52:01 -0700
* Update github actions to checkout v3 and upload-artifact v4 (Tim Wojtulewicz, Corelight)
3.0.0-3 | 2024-02-20 10:42:47 -0700
* Update some documentation about bro-pkg.meta (Tim Wojtulewicz, Corelight)
* Revert "Remove references to bro-pkg.meta and bro-pkg.index" (Tim Wojtulewicz, Corelight)
This reverts commit 964b0e7235febd17e414134737513d600673e77c.
3.0.0 | 2024-02-15 09:21:49 -0500
* Release 3.0.0.
2.13.0-72 | 2024-02-15 09:21:42 -0500
* Revert --ignore-dirty-git addition (Arne Welzel, Corelight)
This reverts commit e3de7c2dd70afde8e5512bdb608ec16138aaf525.
2.13.0-70 | 2024-01-24 12:27:36 -0800
* Set language in conf.py to avoid Sphinx warning (Christian Kreibich, Corelight)
* Incorporate sphinx-argparse fix for deprecation warnings in regex. (Christian Kreibich, Corelight)
2.13.0-67 | 2024-01-24 09:31:19 -0800
* Add developer docs about zkg's internal use of directories (Christian Kreibich, Corelight)
2.13.0-64 | 2023-11-06 12:49:34 +0100
* manager/package: Use is_valid_name() for alias validation (Arne Welzel, Corelight)
Tighten is_valid_name() to disallow filesystem separator and
leading dots.
* GH-168: manager: Check for alias conflicts (Arne Welzel, Corelight)
Prevent aliases to shadow installed packages, or newly installed
packages to replaces aliases, or the same aliases from different
packages clashing.
* _parse_package_metadata: Restrict alias names (Arne Welzel, Corelight)
Ensure aliases do not contain overly surprising characters. This is
mainly to avoid directory traversals because it can be used to
delete existing symlinks. Also, os.path.join("/a/b", "/c") results
in "/c", so an alias of "/c" would place a symlink in /.
* GH-171: manager: Support __zeek_plugin__ magic file (Arne Welzel, Corelight)
Zeek 6.1 switched from __bro_plugin__ to __zeek_plugin__. Modify
_write_plugin_magic() to handle the new and the old file.
* overview: Replace :note: with note directive (Arne Welzel, Corelight)
Looks better and is what we use elsewhere.
* doc/conf.py: Use sphinx_rtd_theme also when on_rtd (Arne Welzel, Corelight)
Do not rely on it being set implicitly by readthedocs. It's not anymore.
* Place .readthedocs.yaml (Arne Welzel, Corelight)
* testing: Bump minimum required cmake version rot13 test plugin (Arne Welzel, Corelight)
* Fix Github edit links in top-right of HTML pages (Christian Kreibich, Corelight)
These links 404'd because they omitted the file suffix, and they didn't really
link to anything editable, just the raw file.
---
In this case, "suffix" was used which is set by the readthedocs theme in
older versions [1]. Seem good to replace with page_source_suffix.
[1] https://github.com/readthedocs/sphinx_rtd_theme/pull/1104
2.13.0-51 | 2023-07-17 18:07:42 -0700
* CI: automatically push to PyPI when pushing a git tag (Christian Kreibich, Corelight)
* CI: add Github workflow for testing zkg (Christian Kreibich, Corelight)
* CI: remove Cirrus setup (Christian Kreibich, Corelight)
* Don't analyze __bro_plugin__ file content in tests, only its presence (Christian Kreibich, Corelight)
2.13.0-46 | 2023-06-30 12:47:34 -0700
* Update man page with changes to other documentation (Tim Wojtulewicz, Corelight)
* Remove references to bro-pkg.meta and bro-pkg.index (Tim Wojtulewicz, Corelight)
* Remove references to legacy bro naming (Tim Wojtulewicz, Corelight)
This intentionally leaves behind references to bro-pkg.meta and __bro_plugin__
but removes all of the other references to scripts and files that have been
deprecated and/or removed since Zeek 3.0.
* User-visible warning when bro-pkg.meta is in use. (Arne Welzel, Corelight)
Relates to #119
* bundle/unbundle: Rework built-in dependency check (Arne Welzel, Corelight)
Extend unbundle to validate dependencies of contained packages and log a
warning if any are not fulfilled or fail to validate. This can happen when
a bundle is created with --nodeps or the target system has different
built-in packages as compared to the source system.
Built-in packages are included in a bundle's manifest.txt file in a new
[meta] section.
* manager: Do not include built-in packages in manifest (Arne Welzel, Corelight)
Bugfix and test for mistakenly recording built-in packages within
the manifest.json file. The built-in information is only ever
requested from Zeek.
* package: Add PackageVersion (Arne Welzel, Corelight)
Make checking if a package version fullfills a version_spec re-usable.
This certainly changes the messages produced, but doesn't seem there
are many tests upset about that.
* Add built-in package awareness (Arne Welzel, Corelight)
This change uses Zeek 6.0's --build-info flags to discover the "zkg.provides"
entry and makes ``zkg`` aware of these as installed packages.
It introduces the following changes
* A pseudo zeek-builtin:// scheme for git_urls to recognize when a
Package instance relates to a built-in package
* The package source name "zeek-builtin" is now reserved
* A zkg bundle's manifest now contains a new section called `[bundle_builtin]`
listing any packages that are expected to exists as built-in packages on
the target system. The content is derived from the built-in packages on
the system building the bundle. This is somewhat narrow and opinionated,
but at the same time avoids the complexity of any constructed scenarios
one can come up with here.
* A new tracking method "builtin" exists. It should act as version but
makes it explicit that a package is built-in.
Most commands have been extended to either ignore (purge) or fail
(install, remove, test, load, unload, pin, unpin) when they detect
a built-in package being operated on. The commands list and info
received a ``--include-builtin`` option for explicitly including
built-in packages in the output.
* zkg: Log warnings by default (Arne Welzel, Corelight)
Currently, to see warnings generated by zkg a user has to pass a single -v.
Change this behavior by always installing the StreamHandler() with a
warning level so that warnings are displayed by default.
* package: Add a few __repr__() implementations (Arne Welzel, Corelight)
Not having __repr__() implemented makes it more difficult than needed
when using print / IPython shell to look around and discover state.
Implement some of them to aid that development style.
They aren't perfect, but better than not having them at all.
* _util: Add ZeekInfo helper class (Arne Welzel, Corelight)
Mostly for collecting executable paths for now, but could also see
the ZEEKPATH collections being done there down the road.
2.13.0-32 | 2023-06-20 15:55:21 -0700
* Add --ignore-dirty-git option to test and install commands (Tim Wojtulewicz, Corelight)
2.13.0-30 | 2023-04-21 20:37:27 +0200
* Reduce to 2 CPUs per task for CI tasks (Arne Welzel, Corelight)
Specifying 8 CPUs and two tasks immediately allocates as many resources
as are available for free for a single user on Cirrus. Depending on what
else a user is using Cirrus for, execution of zkg tasks may be delayed
for significant amounts of time.
The test-suite does not run such a long duration that the 4x reduction
in CPUs should have a practical impact.
* Fix upgrade using test_command of old version (Arne Welzel, Corelight)
* GH-137: Change --force to fail on failing tests (Arne Welzel, Corelight)
This is a user-visible change in behavior, but to something sane, so
hopefully that's acceptable.
Currently, `--force` is oftentimes used in scenarios where it should mean
"non-interactive". At the same time, `--force` also implies to ignore
failing test commands. This is not reasonable behavior, particularly as
`--skiptests` is available. This patch changes the behavior and causes
install/upgrade to exit with a failure code when tests fail even with
`--force` provided.
If someone intentionally wants to install a package with broken tests,
they can/should use `--skiptests` explicitly instead. That is less
surprising and stands out more.
Fixes #137
2.13.0-26 | 2023-02-03 09:23:59 +0100
* Fix zkg bundled usage (Arne Welzel, Corelight)
This reverts commit 2b11486b09d76df40aafd411aae88cb5cf902850 which
naively moved imports above a sys.path.insert()...
Adapt flake8 to ignore those import warnings and add a test so hopefully
this won't happen again.
2.13.0-24 | 2023-01-17 14:29:45 -0800
* setup.cfg/pre-commit-config: Add flake8 hook and config (Arne Welzel, Corelight)
* zkg: Fix comparison with True (Arne Welzel, Corelight)
* zkg: Fix unused variables (Arne Welzel, Corelight)
* zkg: Imports at top of file (Arne Welzel, Corelight)
* manager: Bad logging invocatoins (Arne Welzel, Corelight)
* Remove CodeQL (Christian Kreibich, Corelight)
2.13.0-17 | 2023-01-10 12:54:54 -0800
* Replace manual nested loops with set operation. (Benjamin Bannier, Corelight)
* Make sure files are properly closed in `setup.py`. (Benjamin Bannier, Corelight)
* Automatically update Python sources to use python-3.7 syntax. (Benjamin Bannier, Corelight)
* Add `.git-blame-ignore-revs` file. (Benjamin Bannier, Corelight)
* Reformat Python code with Black. (Benjamin Bannier, Corelight)
* Add GH action to run pre-commit. (Benjamin Bannier, Corelight)
* Add pre-commit config. (Benjamin Bannier, Corelight)
2.13.0-9 | 2023-01-05 13:17:20 -0800
* Tweaks to appease CodeQL (Christian Kreibich, Corelight)
* Add CodeQL workflow for GitHub code scanning (LGTM Migrator)
2.13.0-5 | 2022-11-03 19:56:59 -0700
* Fix git complaint about unsafe use of file protocol in submodule test (Christian Kreibich, Corelight)
2.13.0-3 | 2022-10-27 16:05:46 -0700
* Add safe wrapper around tarfile.extractall() (Christian Kreibich, Corelight; Charles McFarland, Trellix)
* Newline whitespace tweaks for consistency, no actual change (Christian Kreibich, Corelight)
2.13.0 | 2022-06-01 01:18:25 -0700
* Release 2.13.0.
2.12.0-22 | 2022-06-01 01:18:08 -0700
* Add argcomplete support (Christian Kreibich, Corelight)
2.12.0-20 | 2022-04-28 14:12:23 -0700
* Adjust requirements.txt to use Sphinx 3+, and drop Napoleon (Christian Kreibich, Corelight)
* Switch to the sphinx-bundled version of the Napoleon add-on (Christian Kreibich, Corelight)
2.12.0-17 | 2022-04-25 10:36:42 -0700
* Add test to verify progress indicator behavior on TTYs (Christian Kreibich, Corelight)
* Remove diff-remove-install-ticks canonifier (Christian Kreibich, Corelight)
* Add InstallWorker.wait() method and switch progress-dot writers to it (Christian Kreibich, Corelight)
2.12.0-13 | 2021-11-04 16:39:58 -0700
* Request at least Sphinx 2.0 to avoid a dependency problem in RTD (Christian Kreibich, Corelight)
2.12.0-11 | 2021-11-01 11:20:36 -0700
* Expand initializer to set up repos also for our test templates (Christian Kreibich, Corelight)
* Improve template source info in fresh zkg.meta (Christian Kreibich, Corelight)
The zkg.meta of freshly created packages so far just said "source = <name>",
where <name> derived from the template's storage location (e.g. just
"package-template" for the standard template). We now report the origin URL if
the template comes with a git repo.
* Show precise commit info in template commit when instantiating from branch (Christian Kreibich, Corelight)
We previously just showed something like "version = master", which won't be
helpful over time. We now incude the exact commit when not installing from a
specific commit or tag.
* Remove "zeek-config --zeek_dist" requirement from two tests (Christian Kreibich, Corelight)
* Modernize cmake requirements in the rot13 plugin-package (Christian Kreibich, Corelight)
* Re-enable CI for the feature releases and use new Dockerfile (Christian Kreibich, Corelight)
* Rework Dockerfiles to using our binary packages (Christian Kreibich, Corelight)
* Add TEST-REQUIRES for zeek-config to three tests that require it (Christian Kreibich, Corelight)
2.12.0 | 2021-10-12 13:58:09 -0700
* Release 2.12.0.
2.11.0-18 | 2021-10-12 13:57:17 -0700
* Support testing package dependencies in Manager.test() (Christian Kreibich, Corelight)
* Interpolate test_command (Christian Kreibich, Corelight)
* Strengthen the installation-staging concept (Christian Kreibich, Corelight)
* Update baseline of test affected by new package install order (Christian Kreibich, Corelight)
* Consistently modify the environment during staging and testing (Christian Kreibich, Corelight)
* Install/upgrade packages in reverse dependency order (Christian Kreibich, Corelight)
* Ensure breadth-first reporting of new nodes in Manager.validate_dependencies() (Christian Kreibich, Corelight)
* Install executables during staging (Christian Kreibich, Corelight)
2.11.0-4 | 2021-08-16 14:04:35 -0700
* Accept `uninstall` as an alias for `remove`. (Benjamin Bannier, Corelight)
* Incorporate sphinx-argparse upstream fix for aliased commands (Christian Kreibich, Corelight)
2.11.0 | 2021-07-05 20:58:29 -0700
* Release 2.11.0.
2.10.0-10 | 2021-07-05 20:53:39 -0700
* Require zeek-config for two additional tests that will fail without it (Christian Kreibich, Corelight)
* Add --fail-on-aggregate-problems to the refresh command (Christian Kreibich, Corelight)
When provided (in addition to --aggregate), any package metadata processing
problems cause zkg to exit with error. Without the flag, such problems only
trigger a warning.
* Added resilience when specifying local paths as package git repos (Christian Kreibich, Corelight)
In addition to failing when a given path doesn't exist, the git.Repo()
constructor also fails when a given path exists but is not a git
repository. That scenario caused backtraces with package installs,
testing, and bundling when providing such invalid local paths.
We now check for this case and provide according output. This adds
guards for the install, test, and bundling commands, which are the
entry points for providing local URLs.
This also expands the test case for invalid repos.
* Fix crash in case the user enters just "zkg template" (Christian Kreibich, Corelight)
The output of "zkg template" (which will gain various commands over
time) now mirrors that when just saying "zkg". Also, the help output
in `zkg template --help` no longer double-mentions "template".
2.10.0 | 2021-06-21 19:51:19 -0700
* Release 2.10.0.
2.9.0-10 | 2021-06-21 19:08:22 -0700
* When zkg is bundled with Zeek, prepend Zeek's Python module path (Christian Kreibich, Corelight)
Prepending instead of appending ensures that the bundled zkg picks the
zeekpkg module shipped with Zeek, not any others available elsewhere
in Python's search path.
* Improve the package source refresh test (Christian Kreibich, Corelight)
This test now also verifies the "refresh --push" behavior. To do this
it makes the "remote" package source bare and verifies the log zkg
generates at the debug level, with a bit of additional canonification.
For better output filtering, the diff-remove-abspath canonifier now
recognizes whitespace (it's the same as used by Zeek), and
diff-remove-zkg-version now actually filters the current zkg version,
not just a related token that appeared in one test.
* Add logging to package source refreshes (Christian Kreibich, Corelight)
At log level INFO zkg now reports whether the local aggregate.meta was
dirty during a "refresh --push" and thus led to a commit; at log level
DEBUG it also reports the package additions/changes/drops that end up
in such a commit.
* Harden "@" interpretation in package source URLs (Christian Kreibich and Arne Welzel, Corelight)
urlparse parses scp-style git URLs ("[email protected]:foo/bar") crudely
as path-only URLs, exposing us to a risk of confusing user@ prefixes
with @version suffixes. We now look for scp-style URLs and transform
those explicitly to the "ssh://" schema to disambiguate.
2.9.0-1 | 2021-05-27 13:46:33 -0700
* Update Python requirement to 3.6+ (Jon Siwek, Corelight)
Some new code that came in with zkg 2.9.0 does indeed use Python
3.6 features (f-strings), but also early Python versions are generally
end-of-life already.
2.9.0 | 2021-05-18 15:07:05 -0700
* Release 2.9.0.
2.8.0-24 | 2021-05-18 15:06:32 -0700
* Add an baseline canonifier to user-mode btest (Jon Siwek, Corelight)
* Change templating API_VERSION to 1.0.0 (Jon Siwek, Corelight)
2.8.0-22 | 2021-05-18 13:13:17 -0700
* Minor docstring formatting improvements (Jon Siwek, Corelight)
* Add 'uservar' and 'template' modules to API docs (Jon Siwek, Corelight)
* Fix sphinxarg extension's use of old Sphinx logging API (Jon Siwek, Corelight)
2.8.0-19 | 2021-05-18 12:03:36 -0700
* Minor developer guide tweaks (Jon Siwek, Corelight)
2.8.0-18 | 2021-04-28 12:21:12 -0700
* Add "template info" command (Christian Kreibich, Corelight)
"template info" shows information about the provided template (or
the default), in either JSON or plaintext. This verifies whether
a given template is loadable and reports supported features,
user vars, and tagged versions.
* Provide default branch name in our git wrapper (Christian Kreibich, Corelight)
Newer git versions trigger a warning when the default branch name is
not set, which cluttered up the logs. This adds the default name,
preserving "master" since that branch name is currently visible in
test baselines.
* Add templating functionality to zkg toplevel script (Christian Kreibich, Corelight)
- Provide "zkg create" to instantiate a template, with optional
features and user vars for parameterization.
- ZKG_DEFAULT_TEMPLATE points at the default template repo,
https://github.com/zeek/package-template.
* Add a "slug" helper for user vars (Christian Kreibich, Corelight)
For templates it'll often be handy to render a variable value into a
file-system-safe identifier ("slug"). This provides uservar.slugify()
for this purpose.
* Add Zeek package templating module (Christian Kreibich, Corelight)
This provides infrastructure for structuring and instantiating Zeek
package templates. Templates are standalone git repos, managed as
follows:
- Template objects define package templates, including an output
directory, a specific package template, any additional features, and
parameters substituted when instantiating. Parameters derive from user
variables in a way defined by the template.
- The Package and Feature classes model template content and allow
validation of input parameters and instantiation of content. The
package class has additional functionality for initializing new
packages as git repos and adding template metadata to its zkg.meta.
- A hierarchy of exceptions accommodates template-specific problems.
These aren't currently imported into the zeekpkg namespace -- you need
to use zeekpkg.template.* when accessing those components.
* Add _util.load_source() for sourcing a Python script as a module (Christian Kreibich, Corelight)
Templates are driven by a toplevel __init__.py script, which we source
via this function.
* Prefill the prompt for user vars with current value (Christian Kreibich, Corelight)
This preserves the current model of allowing a quick hit of enter to
confirm the existing value, but also lets you empty out the value to
provide an empty string as the new value.
* Refactor user vars into their own class and module (Christian Kreibich, Corelight)
- Moves user-var functionality to a new uservar module and adds
UserVar, a class representing an individual user variable. These can
be created explicitly, parsed from a dict, or parsed from a NAME=VAL
string.
- Adds support for --user-arg NAME=VAL for the install, unbundle, and
upgrade commands. This can be provided repeatedly and lets you provide
user variables via the command line.
- Slightly tweaks the user prompt to accommodate situations other than
package installation.
* Migrate some git helpers to _util for broader access (Christian Kreibich, Corelight)
2.8.0 | 2021-03-26 16:50:59 -0700
* Release 2.8.0.
2.7.1-13 | 2021-03-26 16:50:20 -0700
* Check existence of executables only after the build process. (Robin Sommer, Corelight)
2.7.1-11 | 2021-03-24 18:05:00 -0700
* GH-101: include exception output in dependency import error messages (Jon Siwek, Corelight)
2.7.1-9 | 2021-03-24 15:38:03 -0700
* Add support for packages shipping executables. (Robin Sommer, Corelight)
This introduces a new "executables" metadata field that a package can
set to scripts or binaries that it wants to make available to users. On
package installation, zkg will link these executables into a new,
central "bin_dir" that users can put into their PATH. By default, that's
"<state>/bin", but it can be relocated through a corresponding "bin_dir"
entry in the configuration file. On package removal, these symlinks get
removed from "bin_dir". Users can automatically update their path
through something like PATH=$(zkg config bin_dir):$PATH.
2.7.1-7 | 2021-03-05 16:39:22 -0800
* Update plugin used in test suite for Zeek 4.1 compatibility (Jon Siwek, Corelight)
* Update test baselines to include btest header info (Jon Siwek, Corelight)
2.7.1-5 | 2021-03-05 15:56:09 -0800
* Prevent local githooks from interfering with zkg's commands (Christian Kreibich, Corelight)
This adds --no-verify during git commits and pushes to prevent githook
interference.
* Prevent local githooks from interfering with the testsuite (Christian Kreibich, Corelight)
This adds a git wrapper script that skips any git configuration done in
the user's home, including typical local git hooks.
* Update CI to test Zeek 4.0 and not test 3.2 anymore (Jon Siwek, Corelight)
2.7.1 | 2021-02-05 15:29:59 -0800
* Release 2.7.1.
2.7.0-2 | 2021-02-05 15:29:41 -0800
* Teach autoconfig command the --force option (Jon Siwek, Corelight)
Allows for skipping confirmation prompts
2.7.0 | 2021-01-25 12:05:08 -0800
* Release 2.7.0.
2.6.1-7 | 2021-01-25 11:54:28 -0800
* Documentation updates for user mode (Christian Kreibich, Corelight)
* Expand use of environment variables to override internal settings (Christian Kreibich, Corelight)
- ZKG_DEFAULT_SOURCE allows setting an alternative default package
source when https://github.com/zeek/packages isn't desirable. An
empty value causes no source to be defined.
- ZEEK_ZKG_CONFIG_DIR and ZEEK_ZKG_STATE_DIR allow overriding internal
storage locations configured during a Zeek-bundled install. This is
mainly to aid testing, so the help output does not include these.
* Suggest use of --user when zkg detects write-permission problems (Christian Kreibich, Corelight)
* Add user mode to zkg (Christian Kreibich, Corelight)
The --user flag forces zkg to manage state, including scripts and
plugin directories, in ~/.zkg. --user is mutually exclusive to
--configfile, and also overrides any ZKG_CONFIG_FILE environment
variable.
The flag exists for all commands, so you can also use e.g. autoconfig
to write out a config file with those settings. As before, "zkg env"
reports the environment variables required for operating Zeek with
these settings.
* Avoid reporting repeated paths in "zkg env" (Christian Kreibich, Corelight)
2.6.1 | 2021-01-06 21:35:39 -0800
* Release 2.6.1.
2.6.0-8 | 2021-01-06 21:33:22 -0800
* Remove unneeded Python module imports (Jon Siwek, Corelight)
* Remove excess argument to a string format() (Jon Siwek, Corelight)
* Remove an unreachable return statement (Jon Siwek, Corelight)
2.6.0-5 | 2021-01-06 21:02:09 -0800
* Fix missing argument to a function in dependency analysis logic (Jon Siwek, Corelight)
2.6.0-4 | 2021-01-06 20:52:13 -0800
* When installed as a subproject of Zeek, automatically add its bin/ to PATH (Christian Kreibich, Corelight)
This helps package installations succeed that require executables from
that directory, such as zeek-config.
2.6.0 | 2020-12-12 21:20:21 -0800
* Release 2.6.0.
2.5.0-12 | 2020-12-12 21:18:34 -0800
* Update quickstart docs to note zkg comes installed with Zeek 4.0.0+ (Jon Siwek, Corelight)
* Install zkg.1 man page as part of CMake builds (Jon Siwek, Corelight)
2.5.0-10 | 2020-12-12 20:21:32 -0800
* Support cmake-driven installation of zkg when bundled with Zeek (Christian Kreibich, Corelight)
This adds cmake-level templating of the toplevel zkg script to substitute
paths that let zkg find its own module. When installing independently,
this mechanism has no effect, since the zeekpkg module continues to be
found via usual PYTHONPATH mechanisms.
The templating also lets us adjust the default location of the config
and state directories, since in Zeek-bundled installs they are
separate.
2.5.0-8 | 2020-12-10 15:25:50 -0800
* Add explanatory error message to zkg for failed imports of dependencies (Jon Siwek, Corelight)
If the external gitpython or semantic-version dependencies aren't
available when running `zkg`, it now shows an error messages to explain
what's required and an example of how to install them.
2.5.0-7 | 2020-12-07 17:19:11 -0800
* Replace remaining use of Thread.isAlive with Thread.is_alive (Christian Kreibich, Corelight)
isAlive() has been deprecated for a while. Python 3.9 (Fedora 33's
default) no longer includes it, which broke the metadata-depends and
metadata-suggests tests.
2.5.0-4 | 2020-12-04 12:43:42 -0800
* Remove deprecated bro-pkg script and bropkg module (Jon Siwek, Corelight)
The PyPI bro-pkg package will also no longer be kept in sync with zkg.
* Simplify configparser imports/usages (Jon Siwek, Corelight)
* Remove future print_function imports (Jon Siwek, Corelight)
* Remove configparser from requirements.txt (Jon Siwek, Corelight)
2.5.0 | 2020-12-04 10:30:29 -0800
* Release 2.5.0.
2.4.2-9 | 2020-12-04 10:28:22 -0800
* Extend test suite to cover a package with a "main" default branch (Jon Siwek, Corelight)
As opposed to historical default of "master" being the convention.
* GH-76: Detect default branch name of packages automatically (Jon Siwek, Corelight)
Rather than use the hardcoded 'master' branch as the conventional
default, zkg will now attempt to select a package's default branch as
follows:
If there is a remote named 'origin' and it has a HEAD reference,
whatever branch that points to is taken to be the default branch.
If there is not a remote named 'origin' or it does not have a HEAD,
then the default branch is selected in this order: 'main' if it exists,
'master' if it exists, the currently active branch (HEAD) if there is
one, else the commit hash of the current detached head.
Note that zkg still prioritized release version tags, like 'v1.0.0',
over branches and the default branch selection logic only applies
when no such release version tag exists.
* Make test cases resilient to non-master default git branch names (Jon Siwek, Corelight)
2.4.2-5 | 2020-11-26 17:38:59 +0000
* Update pip invocations to use explicit `pip3` (Jon Siwek, Corelight)
* Remove Python compatibility logic for versions less than 3.5 (Jon Siwek, Corelight)
* Update Python invocations to use explicit `python3` (Jon Siwek, Corelight)
* Update documentation to reflect new minimum Python 3.5 requirement (Jon Siwek, Corelight)
2.4.2 | 2020-11-10 15:18:54 -0800
* Release 2.4.2.
2.4.1-5 | 2020-11-10 15:17:45 -0800
* Fix/improve dependency resolution failure messages (Jon Siwek, Corelight)
The no_best_version_string() function was incorrectly returning a tuple
instead of a string.
* GH-83: Fix branch-based dependency analysis for packages with no tags (Jon Siwek, Corelight)
Branch-based dependency analysis was incorrectly skipped for packages
that had no release version tags.
* GH-82: Enforce dependency requirements of already-installed packages (Jon Siwek, Corelight)
Treat as an error the installing a package that would break dependency
requirements of an already-installed package.
2.4.1 | 2020-11-03 14:55:16 -0800
* Release 2.4.1.
2.4.0-13 | 2020-11-03 14:52:09 -0800
* Improve "remove" operation to not unload already-unloaded dependencies (Jon Siwek, Corelight)
* Improve how "unload" operation manages dependencies (Jon Siwek, Corelight)
* Added a confirmation prompt for the "unload" operation
and a `--force` option to bypass it
* The confirmation prompt now lists all dependent packages that
will also be unloaded as part of the request
* Rename the "runtime dependency management" btest (Jon Siwek, Corelight)
* GH-78: Improve structure/output of "remove" operations (Jon Siwek, Corelight)
* Previously, the remove operation would indicate that all other
installed packages besides the one being removed were dependent
and needed to be unloaded. No such action would ever happen, it
was just incorrect/extraneous output.
* Previously, a second "Proceed?" prompt would appear (as part of the
logic to ask if it's alright to unload dependers) even if there's
only a single, independent package being removed.
* With this change, the entire set of depender-packages is gathered
up front to display both packages-to-remove and packages-to-unload
within the a single "Proceed?" prompt. This makes it clear what the
overall results of the remove operation will be right away rather than
prompt per package-to-remove asking just-in-time if it's ok to unload
dependers. That is, easier for user to make a single decision at the
start rather than find out midway that something isn't to their liking
and abort then, possibly leaving things in an undesirable state that
still needs sorting out.
This is also helps streamline the final output of the resulting
unload/remove operations which was previously hard to parse since it
included redundant information about which were unloaded and also
unimportant errors like saying a package couldn't be unloaded at a
certain point because it was still in use by another package except
that other package was later scheduled to also be unloaded, making
that error moot.
* Fix various docstring typos/mistakes (Jon Siwek, Corelight)
* GH-79: fix dependency-aware (un)loading logic to ignore "reserved" names (Jon Siwek, Corelight)
Both `Manager.load_with_dependencies()` and
`Manger.unload_with_unused_dependers()` previously could find a package
depending on "zeek" (or "zkg") and consider that as the name of a
package to (un)load as part of the operation and fail since they're not
real packages that are installed. Those operations now simply skip over
such reserved names when walking the dependency graph since there's
nothing to do for them.
2.4.0-2 | 2020-11-02 13:38:01 -0800
* Remove superfluous "fetch" during git cloning (Jon Siwek, Corelight)
Fetching tags used to be necessary for shallow clones, but that now
happens implicitly since changing to use `git clone --no-single-branch`
(which is also needed for zkg's branch-based version tracking).
A secondary `git fetch` after doing a shallow clone also has negative
effect of causing errors on older git versions (e.g. v1.8.3.1, currently
found in CentOS 7) since they prohibit fetching from a shallow clone.
2.4.0 | 2020-10-20 15:45:32 -0700
* Release 2.4.0.
2.3.1-2 | 2020-10-20 14:26:32 -0700
* GH-74: handle `git checkout` failures during `zkg refresh --aggregate` (Jon Siwek, Corelight)
Previously, a checkout failure caused the entire aggregation process
to fail, now it will just skip the offending package and later emit
a warning that it could not collect its associated metadata.
2.3.1-1 | 2020-10-20 13:37:16 -0700
* GH-75: show more warnings from `zkg refresh --aggregate` (Jon Siwek, Corelight)
Package metadata collection issues (e.g. missing metadata file) were not
previously surfaced (except when using `zkg -v` verbosity), but these
might always be interesting to know about for those performing the
aggregation themselves.
2.3.1 | 2020-09-25 12:24:47 -0700
* Release 2.3.1.
2.3.0-5 | 2020-09-25 12:22:22 -0700
* GH-70: When running tests, use any already installed dependencies (Robin Sommer, Corelight)
Rather than rebuilding them.
* Small tweak to -v to turn any count >= 3 into debug. (Robin Sommer, Corelight)
* Simplify installation thread "progress ticks" logic (Jon Siwek, Corelight)