Haru

Correct

The 2013-08-07 client falls into this category.

There were reasons why signed was chosen instead of unsigned. Most operations we do with tick are tick differences, and that kind of operation with unsigned variables requires extra care that's not required with signed ones. There really isn't any benefit in using unsigned in most cases (especially if it's just to say that the variable should never be negative), and I strongly believe the only place where one should use unsigned is where the value isn't used as a plain number, but a bit field or similar. If the point about using unsigned is to allow larger values, well, the int64 overflow will happen about 292 million years from now. I believe that, whichever intelligent life forms will be alive by then, if any, will be able to use a different 'zero', or a larger variable. This article on the topic is an interesting read: http://www.soundsoftware.ac.uk/c-pitfall-unsigned

Added: [*]Added support for automatic concatenation of adjacent string literals (09dd209) [*]Improved scripting operators precedence rules (ac6ae8c) [*]Added support for ++x / --x operators and changed the x++ / x-- operators to post-increment (9a802b9) (they were authored in September, but actually pushed in November, so they were accidentally skipped in the digest)

item_db and item_db_re are automatically generated by the HerculesWS API bot. For item_db2 (or if you have edited your item_db for any reason), you can build your custom version with the db2sql plugin. (instructions for linux/unix; if you're on windows, YMMV) - Compile the db2sql plugin (make -C src/plugins db2sql) - Enable it in conf/plugins.conf - Run the map server with ./map-server --db2sql - You can find the updated files in the sql-files folder.

This means your client is not using any obfuscation, because you (or someone for you, if you're not the one who hexed it), disabled it. This means you'll probably be able to connect to the server even if you have packet_obfuscation set to 0. Hercules doesn't currently have the encryption keys for that client version though (unless they're the same as the 20130703 client, but I doubt so), so even if you enable the obfuscation clientside, you might not be able to connect to the server unless you also add the correct keys.

The crash should be solved in https://github.com/HerculesWS/Hercules/commit/d9a89c56ea63e369999ca13f1584eef9e511490f (it was an oversight on my part caused by a last minute change before committing)

Yes. There's a version packaged for easy installation (ActivePerl) at http://www.activestate.com/activeperl/downloads (You can download the free edition)I have just tested it in a windows virtual machine with the version 5.16.3.1603 downloaded from that site, and it works without any issues.

Not necessarily. OS X will work out of the box, and so probably will most UNIX variants. It is likely that it works even on Windows (as long as you install the perl interpreter, since it doesn't come pre-installed)

I'm perfectly fine with backwards compatibility, whenever it doesn't slow down development or force developers to use hacks and workarounds to introduce new features. Can't find a topic. Please send me a link via. PM. Done~ That's the point. Think of the possibilities of SQL. You would be able to change data while server is running and this not just temporary. You could even create completely new data structures if required for a script. at runtime. (You can do this know, but not when dropping SQL.) Wouldn't whis be an advantage? "Hard to read".. Well, there are many many tools which make it easy to create SQL statements if you don't like typing them, that's not a valid reason in my opinion. You can't really change the data while the server is running (without reloading the database), but then you can reload the txt version in the same exact way (and possibly even faster than the SQL version). I don't really understand the 'create new data structures' part. If you mean creating other SQL tables, sure, but it has nothing to do with the item database. If you mean adding fields to the database -- no, you can't do that, since the code that reads from the database is compiled to the server (both for the txt and the SQL version), so you have at the very least to recompile and restart the server. Also, no matter how you format a SQL statement, it can never become as clean and easy to read as the new item database format. It'll always be a bunch of fields next to each others. this is next on the hit list It indeed is Even if file size increases, I have actually benchmarked it and the parse time decreases. Loading and parsing the old format 1000 times (test done on db/pre_re/item_db.txt) takes 95603ms (average of 93 milliseconds each time), while loading and parsing the new format 1000 times (test done on db/pre_re/item_db.conf) takes 57390ms (so an average of 57 milliseconds). The tests were done respectively on the latest revision before the item db update appeared (7d0a63b72f93c05b302c08cfb19d5651cef1cdb8) and the latest revision (d1a635a06f5152b51d0879176f011b5c6e45431c), so they are even biased towards the old version, since it had less features back then (BindOnEquip didn't exist, so it was reading one less field.) Line count isn't relevant to the parse time. The new parser probably uses a more optimized file reading approach (through libconfig), and doesn't need to unnecessarily read fields that are set to their default values (file is bigger, but it actually contains less data.) If a higher line count disturbs you this much, you can flatten the entire database to ONE line, and it will still work. Make sure you add a semicolon after each value. This is a perfectly valid item database, if one is that crazy: item_db: ( { Id: 501; AegisName: "Red_Potion"; Name: "Red Potion"; Type: 0; Buy: 50; Weight: 70; Script: <" itemheal rand(45,65),0; ">; },{ Id: 502; AegisName: "Orange_Potion"; Name: "Orange Potion"; Type: 0; Buy: 200; Weight: 100; Script: <" itemheal rand(105,145),0; ">; },)(but I refuse to understand why anyone would want that) you didnt even put a single bit of effort into trying to figure it out by yourself o_O Read the first post, it is briefly explained there. You need a perl interpreter and some (very little) ability to run a command from the command-line.

That's weird, I tested it on perl 5.16 and perl 5.12, and it fully works (and the documentation also confirms so http://perldoc.perl.org/perl5100delta.html#Named-Capture-Buffers / http://perldoc.perl.org/perlre.html#%28?%27NAME%27pattern%29 ) o.o What OS, out of curiosity? Also, what does this output? Any errors? perl -e 'my $a="it doesnt workn";$a =~ s/(?<a>.*)nt/$+{a}/; print $a;'

It looks like your perl interpreter doesn't support named captures. The script uses those, so it requires at least perl version 5.10 (which was released six years ago). You can either upgrade your perl interpreter, or use the online version.

The issue was because of the extra spaces you had in your lines. I added support for that situation, since the server used to load those correctly even if the spaces weren't supposed to be there. Thanks for the report. It's fixed in https://github.com/HerculesWS/Hercules/commit/d1a635a06f5152b51d0879176f011b5c6e45431c (and if you use the online version, it was updated as well)

It isn't difficult but a load of work, which isn't necessary. I don't think there will be a converter which is compatible with my private projects. The provided converter is to transition from the old to the new format. It's already there.If you need to use the file for any other purpose I'd recommend you to look at the db2sql plugin code, and modify that to suit your needs. All the item db info is available there (including the default values for the missing fields), and you can change the fprintf format strings to suit your needs. I am aware of that. But that's not an excuse for being stuck in the past with unreadable file formats. Tools can be easily edited and/or rewritten. Regular expressions are much harder to use (if not impossible), but that doesn't prevent you from writing a tool to convert the file to whatever format you find more suitable, and back to the libconfig format. The file in the repository is fine as it is, at least when it's updated, diffs are easier to read, and nobody can sneak in nasty, subtle changes that would pass unnoticed. Actually, it's been discussed since May 2013. Feel free to release one. I personally won't, as there's no need for it to be in the repository (other than making the Hercules database usable with other server emulators, which is outside our scope.) On a side note, the format has now new (optional) fields (see BindOnEquip). Those can't be converted to the "old" format, unless you add another field to it (but then it won't be the same old format anymore - that was one of its biggest weaknesses, it was impossible to add fields in a sane way without rewriting the entire file) Yep. Hated the comma separated version (for anything other than machine parsing) since I saw it the first time. And as fields got added to it, it became harder and harder. I suggested this many times. Noone wants it. And as long as there are just a few files available in SQL my tools will use the TXT version, simply because I don't want to implement a logic for SQL an another one for TXT. There's a reason not to do that. We'd be back to a hard to read file format (the import scripts). I personally find it pointless to have the item db in SQL (other than for control panels and such), because, well, it's an info that gets read once on server startup, and cached in any case. SQL is useful if you only need to look up parts of it (like a control panel does), but quite pointless (and an annoyance to keep up to date) if all you need is to SELECT * on server startup.If one of the two formats were to go away, to me it'd be SQL. But it's gonna stay, mostly for Control Panel purposes. Well, now you have to search another item if the item you are editing inherits another item just to see the values. And perhaps this item inherits anotherone and you now have to search this now. Furthermore you don't see all the values with one view... I my opinion this new format is pretty confusing. You misunderstood the 'Inherit' point. Please re-read how it works.Inherit can only be used in the item_db2 to inherit the values defined in item_db for *the same item*. You can't have inheritance chains, as it's only a boolean. If your custom Knife[3] has 'Inherit: true' set, and you only define a script (i.e. because you want it to have 100 Atk instead of its standard value), it means it'll be exactly a Knife[3], with all other standard parameters from the item_db, and only the Atk you have overridden will be changed. This means that, if for some reason the official item is updated to have a different weight, you wouldn't need to edit yours, as the change will be propagated (unless you override it.)

Item DB file structure overhaul Hello~! Uguu~?! We noticed that the Item Database file format, unchanged for years, is less than optimal (read: terrible) The file is really hard to read (is it the fifteenth or the sixteenth zero? No wait, that line has an extra comma!!) Whenever you merge an update, if you had a customized entry, you're sure to encounter large conflicts that won't be trivial to solve How do we fix it? We're switching to a brand new, modern, file format, making use of the libconfig library we're already using for other configuration files. It uses libconfig. This means the parser is more solid and, perhaps not faster, but surely easier to maintain, with simpler code. And the file format is something you're already used to, since it's the same as many other configuration files we use! Empty fields and the long sequences of those hard to count commas are gone! You just specify the fields you need, and the others can be completely skipped. The item_db2 entries can be left incomplete and set to inherit the original item_db entry. If you have a custom script for your Knife[3], you can just write the script in your item_db, and let it read the other values from the item_db, so that if we update them, you get the update automatically Item scripts can be split into several lines, so they can made easier to read, especially the long ones. We can finally add more fields (to support new features) to the file at any time, easily and without having to edit all the lines (or force you to edit all the lines of your custom item_db2)! Pre-Renewal and Renewal Item databases now use the same format. This also means that you can make use of the min/max level feature in both renewal and pre-renewal (of course, pre-renewal servers will ignore the Matk field, if you specify it, since it's meaningless there) What does it look like? Each entry follows this structure: { // =================== Mandatory fields =============================== Id: ID (int) AegisName: "Aegis_Name" (string, optional if Inherit: true) Name: "Item Name" (string, optional if Inherit: true) // =================== Optional fields ================================ Type: Item Type (int, defaults to 3 = etc item) Buy: Buy Price (int, defaults to Sell * 2) Sell: Sell Price (int, defaults to Buy / 2) Weight: Item Weight (int, defaults to 0) Atk: Attack (int, defaults to 0) Matk: Magical Attack (int, defaults to 0, ignored in pre-re) Def: Defense (int, defaults to 0) Range: Attack Range (int, defaults to 0) Slots: Slots (int, defaults to 0) Job: Job mask (int, defaults to all jobs = 0xFFFFFFFF) Upper: Upper mask (int, defaults to any = 0x3f) Gender: Gender (int, defaults to both = 2) Loc: Equip location (int, required value for equipment) WeaponLv: Weapon Level (int, defaults to 0) EquipLv: Equip required level (int, defaults to 0) EquipLv: [min, max] (alternative syntax with min / max level) Refine: Refineable (boolean, defaults to true) View: View ID (int, defaults to 0) Script: <" Script (it can be multi-line) "> OnEquipScript: <" OnEquip Script (can also be multi-line) "> OnUnequipScript: <" OnUnequip Script (can also be multi-line) "> // =================== Optional fields (item_db2 only) ================ Inherit: true/false (boolean, if true, inherit the values that weren't specified, from item_db.conf, else override it and use default values) }, Here's a Red Potion in the old format: 501,Red_Potion,Red Potion,0,50,,70,,,,,0xFFFFFFFF,7,2,,,,,,{ itemheal rand(45,65),0; },{},{} And here's the same Red Potion in the new format: { Id: 501 AegisName: "Red_Potion" Name: "Red Potion" Type: 0 Buy: 50 Weight: 70 Script: <" itemheal rand(45,65),0; "> }, Not convinced yet it's easier to read? Okay, let's try a Poison Bottle: 678,Poison_Bottle,Poison Bottle,2,5000,,100,,,,,0xFFFFFFFF,7,2,,,,,,{ if(Class==Job_Assassin_Cross) { sc_start SC_DPOISON,60000,0; sc_start SC_ATTHASTE_INFINITY,60000,0; } else percentheal -100,-100; },{},{} changes to: { Id: 678 AegisName: "Poison_Bottle" Name: "Poison Bottle" Type: 2 Buy: 5000 Weight: 100 Script: <" if(Class==Job_Assassin_Cross) { sc_start SC_DPOISON,60000,0; sc_start SC_ATTHASTE_INFINITY,60000,0; } else percentheal -100,-100; "> }, Better, isn't it!? Let's now try a Jellopy: 909,Jellopy,Jellopy,3,6,,10,,,,,,,,,,,,,,{},{},{} Count the commas! Did I miss any? An extra comma you say? No, I don't want to count them, thanks... { Id: 909 AegisName: "Jellopy" Name: "Jellopy" Buy: 6 Weight: 10 }, Not even a comma! Now, help me read what this item does? 13307,Krieger_Huuma_Shuriken1,Glorious Shuriken,4,20,,0,55,,1,0,0x02000000,7,2,34,4,80,1,22,{ bonus2 bAddRace,RC_DemiHuman,95; bonus2 bIgnoreDefRate,RC_DemiHuman,20; bonus bMatkRate,15; autobonus "{ bonus2 bSkillAtk,NJ_HUUMA,100; bonus2 bSkillAtk,NJ_ISSEN,100; }",50,10000; bonus bUnbreakableWeapon,0; if(getrefine()>5) { bonus2 bAddRace,RC_DemiHuman,(getrefine()-3)*(getrefine()-3); bonus2 bIgnoreDefRate,RC_DemiHuman,5; } if(getrefine()>8) { bonus5 bAutoSpellOnSkill,NJ_ISSEN,AL_HEAL,10,1000,1; bonus4 bAutoSpellOnSkill,NJ_HUUMA,NPC_CRITICALWOUND,2,200; } },{},{} { Id: 13307 AegisName: "Krieger_Huuma_Shuriken1" Name: "Glorious Shuriken" Type: 4 Buy: 20 Atk: 55 Range: 1 Job: 0x02000000 Loc: 34 WeaponLv: 4 EquipLv: 80 View: 22 Script: <" bonus2 bAddRace,RC_DemiHuman,95; bonus2 bIgnoreDefRate,RC_DemiHuman,20; bonus bMatkRate,15; autobonus "{ bonus2 bSkillAtk,NJ_HUUMA,100; bonus2 bSkillAtk,NJ_ISSEN,100; }",50,10000; bonus bUnbreakableWeapon,0; if(getrefine()>5) { bonus2 bAddRace,RC_DemiHuman,(getrefine()-3)*(getrefine()-3); bonus2 bIgnoreDefRate,RC_DemiHuman,5; } if(getrefine()>8) { bonus5 bAutoSpellOnSkill,NJ_ISSEN,AL_HEAL,10,1000,1; bonus4 bAutoSpellOnSkill,NJ_HUUMA,NPC_CRITICALWOUND,2,200; } "> }, Which one did you find easier to read? Old or new format? Want to see an example that specifies both min and max levels? Here you go: { Id: 2819 AegisName: "Swordman_Manual" Name: "Swordman Manual" Type: 5 Buy: 0 Weight: 100 Job: 0x00000001 Upper: 47 Loc: 136 EquipLv: [1, 12] Refine: false Script: <" bonus bMaxSP,100; skill SM_BASH,1; skill SM_PROVOKE,1; skill SM_MAGNUM,1; "> }, Okay, one last example. Let's try to make an item_db2 entry for a Pupa Card that gives a bonus of 1000 HP rather than just 700. { Id: 4003 Inherit: true Script: <" bonus bMaxHP,1000; "> }, Done. No need to rewrite the name, location, prices... those are already in the item_db! But... I have several custom items, do I have to manually convert all of them...? Of course you do! No, I'm kidding, don't hate me It's true that you need to convert your item database to the new format, but we can do it for you! Go to http://haru.ws/hercules/itemdbconverter/ and paste (or upload) your item_db2.txt (or even your item_db.txt if you have custom entries there), press the Convert button, wait a few seconds and you're done! It's already converted for you. Easy, isn't it? Don't trust us? No, no, we don't need your custom items, you can sleep safe... But if you still don't want to paste anything on a website... well, we have provided the source code of the converter script! It's in the 'tools' folder of the Hercules repository. All you need is a Perl interpreter (and if you're running Linux or Mac OS, on either your server or your own computer, it's almost certain that you already have that). All you have to do is run it (example: perl tools/itemdbconverter.pl < db/item_db2.txt > db/item_db2.conf), and your item database will be converted in a split second! What if I was using SQL item databases? Well... Then you won't have to worry about the txt databases, unless you were creating the items in txt, then converting them to sql (if you're not doing this, maybe you should consider it, you know? It's easier to create and manage them in txt even if you use the sql version!) If you were converting your custom item databases from the txt version to sql through the db2sql plugin, there are good news for you! We just updated the plugin, and now it's able to create a .sql script rather than just inserting the entries into your database. This means that you can create the SQL entries in your own computer, without even needing a database (just a copy of Hercules), without risking to slow down your server, and then upload them whenever you want. And if you're relying on the .sql scripts we provide... well, there are still good news for you! Outdated item_db.sql files are a thing of the past now: those scripts will get updated automatically after every commit by our HerculesWS API bot, exactly like the HPM Hooks! As a final note, now the pre-re and re databases share the same structure in SQL as well! no more column mismatch if you're trying to import into your Renewal server an item that was meant for pre-renewal. If you have data already imported to your item_db2 SQL table, it'll be updated to the new table format through the regular upgrade sql mechanism (just run the provided script in the sql-files/upgrades folder). Please note that the script requires MySQL 5.0 or newer -- if you're still on MySQL 4.0, please open it in a text editor, read the comments and run the provided queries manually. I have this event item entry that came with an old script I downloaded... No worries, you can get it converted. Use the same script (or the provided web page) you'd use to convert an entire item database, it'll work just fine even for an item or two! Special thanks To Ind, for bringing up the idea and pushing it forward until it was implemented. And for his awesome work on the HerculesWS API and database converter plugin. To Yommy, for the original idea, and for some invaluable help finalizing the actual database format. Links~u! Main commit. Removed redundant item_db2_re.sql (now both have the same structure). db2sql plugin update. item_db2 inheritance. item_db2 SQL upgrade script.

This is because:

Is your server up to date? Could this be related to bug 7540 that was fixed last week perhaps?

Is it fixed now after this commit?

How would a full rewrite help preventing these attacks? I feel that ditching an old (but maintained) and well tested codebase to rewrite it from scratch, would have the opposite effect and risk creating more security vulnerabilities, other than breaking compatibility with any old addons and themes. As someone once said, if it ain't broke, don't fix it.

Yes, the 2012-04 clients (and possibly a few others) have their keys chosen in such a way that makes the client effectively always use the same encryption key for each sent packet (thus allowing you to replay a packet as much as you want in WPE.) You can either use a newer client, or edit the keys yourself in both client (you'll have to hex-edit it) and server (edit the appropriate packetKeys line for your PACKETVER in src/map/packets.h and recompile.)