Jump to content
  • Sign Up

Haru

Administrators
 View Profile  See their activity

  • Content Count

    382

  • Joined

  • Days Won

    38

Posts posted by Haru

  2. Posted

    are you sure 2013-08-07 is supported? cu i just checked, wpe is useable even after setting packet obfuscation to 2.

     

    please help!!!

    Yes, the 2012-04 clients (and possibly a few others) have their keys chosen in such a way that makes the client effectively always use the same encryption key for each sent packet (thus allowing you to replay a packet as much as you want in WPE.)

     

    You can either use a newer client, or edit the keys yourself in both client (you'll have to hex-edit it) and server (edit the appropriate packetKeys line for your PACKETVER in src/map/packets.h and recompile.)

    The 2013-08-07 client falls into this category.

  3. Posted

    There were reasons why signed was chosen instead of unsigned. Most operations we do with tick are tick differences, and that kind of operation with unsigned variables requires extra care that's not required with signed ones. There really isn't any benefit in using unsigned in most cases (especially if it's just to say that the variable should never be negative), and I strongly believe the only place where one should use unsigned is where the value isn't used as a plain number, but a bit field or similar. If the point about using unsigned is to allow larger values, well, the int64 overflow will happen about 292 million years from now. I believe that, whichever intelligent life forms will be alive by then, if any, will be able to use a different 'zero', or a larger variable.

     

    This article on the topic is an interesting read: http://www.soundsoftware.ac.uk/c-pitfall-unsigned

    pan and AnnieRuru reacted to this

  4. Posted

    Added:

      [*]Added support for automatic concatenation of adjacent string literals (09dd209)

      [*]Improved scripting operators precedence rules (ac6ae8c)

      [*]Added support for ++x / --x operators and changed the x++ / x-- operators to post-increment (9a802b9)

    (they were authored in September, but actually pushed in November, so they were accidentally skipped in the digest)

    Ind and JulioCF reacted to this

  5. Posted

    item_db and item_db_re are automatically generated by the HerculesWS API bot. For item_db2 (or if you have edited your item_db for any reason), you can build your custom version with the db2sql plugin.

     

     

    (instructions for linux/unix; if you're on windows, YMMV)

     

    - Compile the db2sql plugin (make -C src/plugins db2sql)

    - Enable it in conf/plugins.conf

    - Run the map server with ./map-server --db2sql

    - You can find the updated files in the sql-files folder.

  6. Posted

    This means your client is not using any obfuscation, because you (or someone for you, if you're not the one who hexed it), disabled it. This means you'll probably be able to connect to the server even if you have packet_obfuscation set to 0.

     

    Hercules doesn't currently have the encryption keys for that client version though (unless they're the same as the 20130703 client, but I doubt so), so even if you enable the obfuscation clientside, you might not be able to connect to the server unless you also add the correct keys.

    orange and Shio reacted to this

  10. Posted

    Well, backward campability should be a big point in a community project like this, but maybe that's just my opinion. And forcing people to completely rewrite their tools, isn't nice, too. Adding a field to an existing converter logic is much easier than rethinking the complete logic. I really don't know if the advatages of this update are bigger than the disadvantages. For me personally there are no advatages, so you maybe understand why I'm grumbling.

    I'm perfectly fine with backwards compatibility, whenever it doesn't slow down development or force developers to use hacks and workarounds to introduce new features.

     

     

     

    Actually, it's been discussed since May 2013.

    Can't find a topic. Please send me a link via. PM. :)

     

     

    Done~

     

     

    There's a reason not to do that. We'd be back to a hard to read file format (the import scripts). I personally find it pointless to have the item db in SQL (other than for control panels and such), because, well, it's an info that gets read once on server startup, and cached in any case. SQL is useful if you only need to look up parts of it (like a control panel does), but quite pointless (and an annoyance to keep up to date) if all you need is to SELECT * on server startup.

    If one of the two formats were to go away, to me it'd be SQL. But it's gonna stay, mostly for Control Panel purposes.

    That's the point. Think of the possibilities of SQL. You would be able to change data while server is running and this not just temporary. You could even create completely new data structures if required for a script. at runtime. (You can do this know, but not when dropping SQL.) Wouldn't whis be an advantage? "Hard to read".. Well, there are many many tools which make it easy to create SQL statements if you don't like typing them, that's not a valid reason in my opinion.

     

     

    You can't really change the data while the server is running (without reloading the database), but then you can reload the txt version in the same exact way (and possibly even faster than the SQL version).

     

    I don't really understand the 'create new data structures' part. If you mean creating other SQL tables, sure, but it has nothing to do with the item database. If you mean adding fields to the database -- no, you can't do that, since the code that reads from the database is compiled to the server (both for the txt and the SQL version), so you have at the very least to recompile and restart the server.

     

    Also, no matter how you format a SQL statement, it can never become as clean and easy to read as the new item database format. It'll always be a bunch of fields next to each others.

     

     

     

     

    I wish there is also overhaul in Mob db file structure :)

    this is next on the hit list :P

     

     

    It indeed is :D

     

    I like the idea how you guys changed the item database file structure just for the sake that us server owner and you guys as a developers wants a readable format. In my point of view I can only see:

    {	// =================== Mandatory fields ===============================	Id: ID                        (int)	AegisName: "Aegis_Name"       (string, optional if Inherit: true)	Name: "Item Name"             (string, optional if Inherit: true)	// =================== Optional fields ================================	Type: Item Type               (int, defaults to 3 = etc item)	Buy: Buy Price                (int, defaults to Sell * 2)	Sell: Sell Price              (int, defaults to Buy / 2)	Weight: Item Weight           (int, defaults to 0)	Atk: Attack                   (int, defaults to 0)	Matk: Magical Attack          (int, defaults to 0, ignored in pre-re)	Def: Defense                  (int, defaults to 0)	Range: Attack Range           (int, defaults to 0)	Slots: Slots                  (int, defaults to 0)	Job: Job mask                 (int, defaults to all jobs = 0xFFFFFFFF)	Upper: Upper mask             (int, defaults to any = 0x3f)	Gender: Gender                (int, defaults to both = 2)	Loc: Equip location           (int, required value for equipment)	WeaponLv: Weapon Level        (int, defaults to 0)	EquipLv: Equip required level (int, defaults to 0)	EquipLv: [min, max]           (alternative syntax with min / max level)	Refine: Refineable            (boolean, defaults to true)	View: View ID                 (int, defaults to 0)	Script: <"		Script		(it can be multi-line)	">	OnEquipScript: <" OnEquip Script (can also be multi-line) ">	OnUnequipScript: <" OnUnequip Script (can also be multi-line) ">	// =================== Optional fields (item_db2 only) ================	Inherit: true/false           (boolean, if true, inherit the values	                              that weren't specified, from item_db.conf,	                              else override it and use default values)}, 
    35 lines of code if it's a complete code, 3 lines of mandatory code excluding braces. In my item_db.txt i have 8.3k lines of code and I'm still using the old format, converting it will increase the line and yes parsing is not that fast.

     

    I agree it's readable. Format is good but line consuming. Parsing is not that fast. Size of file increase.

     

     

    Even if file size increases, I have actually benchmarked it and the parse time decreases. Loading and parsing the old format 1000 times (test done on db/pre_re/item_db.txt) takes 95603ms (average of 93 milliseconds each time), while loading and parsing the new format 1000 times (test done on db/pre_re/item_db.conf) takes 57390ms (so an average of 57 milliseconds).

     

    The tests were done respectively on the latest revision before the item db update appeared (7d0a63b72f93c05b302c08cfb19d5651cef1cdb8) and the latest revision (d1a635a06f5152b51d0879176f011b5c6e45431c), so they are even biased towards the old version, since it had less features back then (BindOnEquip didn't exist, so it was reading one less field.)

     

    Line count isn't relevant to the parse time. The new parser probably uses a more optimized file reading approach (through libconfig), and doesn't need to unnecessarily read fields that are set to their default values (file is bigger, but it actually contains less data.)

     

    If a higher line count disturbs you this much, you can flatten the entire database to ONE line, and it will still work. Make sure you add a semicolon after each value.

    This is a perfectly valid item database, if one is that crazy:

    item_db: ( { Id: 501; AegisName: "Red_Potion"; Name: "Red Potion"; Type: 0; Buy: 50; Weight: 70; Script: <" itemheal rand(45,65),0; ">; },{ Id: 502; AegisName: "Orange_Potion"; Name: "Orange Potion"; Type: 0; Buy: 200; Weight: 100; Script: <" itemheal rand(105,145),0; ">; },)
    (but I refuse to understand why anyone would want that)

     

     

     

    how to use that sir ?

    you didnt even put a single bit of effort into trying to figure it out by yourself o_O

     

     

    Read the first post, it is briefly explained there. You need a perl interpreter and some (very little) ability to run a command from the command-line.

  14. Posted

     

     

    im not saying its easy but it doesn't look too difficult

    It isn't difficult but a load of work, which isn't necessary.

     

    Dito, but they did promise us a converter~ so let's see bout that

    I don't think there will be a converter which is compatible with my private projects. :(

     

     

    The provided converter is to transition from the old to the new format. It's already there.

    If you need to use the file for any other purpose I'd recommend you to look at the db2sql plugin code, and modify that to suit your needs. All the item db info is available there (including the default values for the missing fields), and you can change the fprintf format strings to suit your needs.

     

    What actually concerns me most is: Did you think of all the tools in use? Every single tool which reads the item DB must be rewritten because of this update.

    Furthermore RegEx replacements are nearly impossible now, too.

    I am aware of that. But that's not an excuse for being stuck in the past with unreadable file formats. Tools can be easily edited and/or rewritten. Regular expressions are much harder to use (if not impossible), but that doesn't prevent you from writing a tool to convert the file to whatever format you find more suitable, and back to the libconfig format. The file in the repository is fine as it is, at least when it's updated, diffs are easier to read, and nobody can sneak in nasty, subtle changes that would pass unnoticed.

    I wish this would have been discussed before implementing, but now it's too late. :S

    Actually, it's been discussed since May 2013.

     

    My suggestion to please everyone: Provide a converter which is capable of converting the new format into the old format.

    Feel free to release one. I personally won't, as there's no need for it to be in the repository (other than making the Hercules database usable with other server emulators, which is outside our scope.)

     

    On a side note, the format has now new (optional) fields (see BindOnEquip). Those can't be converted to the "old" format, unless you add another field to it (but then it won't be the same old format anymore - that was one of its biggest weaknesses, it was impossible to add fields in a sane way without rewriting the entire file)

    Though I get why some would wanna keep using the txt version as it's more convenient. This adds readability to the db, which has been sorely needed for years. Can't tell you the countless times I've had to 'Page Up' to the top part to remember what row / comma section I was in...

    Yep. Hated the comma separated version (for anything other than machine parsing) since I saw it the first time. And as fields got added to it, it became harder and harder.

     

     

     

     

    it would make sense to just start using full sql

    I suggested this many times. Noone wants it. And as long as there are just a few files available in SQL my tools will use the TXT version, simply because I don't want to implement a logic for SQL an another one for TXT.

     

     

    There's a reason not to do that. We'd be back to a hard to read file format (the import scripts). I personally find it pointless to have the item db in SQL (other than for control panels and such), because, well, it's an info that gets read once on server startup, and cached in any case. SQL is useful if you only need to look up parts of it (like a control panel does), but quite pointless (and an annoyance to keep up to date) if all you need is to SELECT * on server startup.

    If one of the two formats were to go away, to me it'd be SQL. But it's gonna stay, mostly for Control Panel purposes.

     

     

    Can't tell you the countless times I've had to 'Page Up' to the top part to remember what row / comma section I was in...

    Well, now you have to search another item if the item you are editing inherits another item just to see the values. And perhaps this item inherits anotherone and you now have to search this now. Furthermore you don't see all the values with one view...  I my opinion this new format is pretty confusing.

     

     

    You misunderstood the 'Inherit' point. Please re-read how it works.

    Inherit can only be used in the item_db2 to inherit the values defined in item_db for *the same item*. You can't have inheritance chains, as it's only a boolean. If your custom Knife[3] has 'Inherit: true' set, and you only define a script (i.e. because you want it to have 100 Atk instead of its standard value), it means it'll be exactly a Knife[3], with all other standard parameters from the item_db, and only the Atk you have overridden will be changed. This means that, if for some reason the official item is updated to have a different weight, you wouldn't need to edit yours, as the change will be propagated (unless you override it.)

    kyeme, Judas and Ind reacted to this

  15. Posted · Edited by Ridley

    Item DB file structure overhaul
     
    Hello~! Uguu~?!
    We noticed that the Item Database file format, unchanged for years, is less than optimal (read: terrible)

    • The file is really hard to read (is it the fifteenth or the sixteenth zero? No wait, that line has an extra comma!!)
    • Whenever you merge an update, if you had a customized entry, you're sure to encounter large conflicts that won't be trivial to solve

    How do we fix it?
    We're switching to a brand new, modern, file format, making use of the libconfig library we're already using for other configuration files.

    • It uses libconfig. This means the parser is more solid and, perhaps not faster, but surely easier to maintain, with simpler code. And the file format is something you're already used to, since it's the same as many other configuration files we use!
    • Empty fields and the long sequences of those hard to count commas are gone! You just specify the fields you need, and the others can be completely skipped.
    • The item_db2 entries can be left incomplete and set to inherit the original item_db entry. If you have a custom script for your Knife[3], you can just write the script in your item_db, and let it read the other values from the item_db, so that if we update them, you get the update automatically
    • Item scripts can be split into several lines, so they can made easier to read, especially the long ones.
    • We can finally add more fields (to support new features) to the file at any time, easily and without having to edit all the lines (or force you to edit all the lines of your custom item_db2)!
    • Pre-Renewal and Renewal Item databases now use the same format. This also means that you can make use of the min/max level feature in both renewal and pre-renewal (of course, pre-renewal servers will ignore the Matk field, if you specify it, since it's meaningless there)

    What does it look like?
     
    Each entry follows this structure:

    {
    // =================== Mandatory fields ===============================
    Id: ID                        (int)
    AegisName: "Aegis_Name"       (string, optional if Inherit: true)
    Name: "Item Name"             (string, optional if Inherit: true)
    // =================== Optional fields ================================
    Type: Item Type               (int, defaults to 3 = etc item)
    Buy: Buy Price                (int, defaults to Sell * 2)
    Sell: Sell Price              (int, defaults to Buy / 2)
    Weight: Item Weight           (int, defaults to 0)
    Atk: Attack                   (int, defaults to 0)
    Matk: Magical Attack          (int, defaults to 0, ignored in pre-re)
    Def: Defense                  (int, defaults to 0)
    Range: Attack Range           (int, defaults to 0)
    Slots: Slots                  (int, defaults to 0)
    Job: Job mask                 (int, defaults to all jobs = 0xFFFFFFFF)
    Upper: Upper mask             (int, defaults to any = 0x3f)
    Gender: Gender                (int, defaults to both = 2)
    Loc: Equip location           (int, required value for equipment)
    WeaponLv: Weapon Level        (int, defaults to 0)
    EquipLv: Equip required level (int, defaults to 0)
    EquipLv: [min, max]           (alternative syntax with min / max level)
    Refine: Refineable            (boolean, defaults to true)
    View: View ID                 (int, defaults to 0)
    Script: <"
        Script
        (it can be multi-line)
    ">
    OnEquipScript: <" OnEquip Script (can also be multi-line) ">
    OnUnequipScript: <" OnUnequip Script (can also be multi-line) ">
    // =================== Optional fields (item_db2 only) ================
    Inherit: true/false           (boolean, if true, inherit the values
                                  that weren't specified, from item_db.conf,
                                  else override it and use default values)
},

    Here's a Red Potion in the old format:

    501,Red_Potion,Red Potion,0,50,,70,,,,,0xFFFFFFFF,7,2,,,,,,{ itemheal rand(45,65),0; },{},{}

    And here's the same Red Potion in the new format:

    {
    Id: 501
    AegisName: "Red_Potion"
    Name: "Red Potion"
    Type: 0
    Buy: 50
    Weight: 70
    Script: <" itemheal rand(45,65),0; ">
},

    Not convinced yet it's easier to read? Okay, let's try a Poison Bottle:

    678,Poison_Bottle,Poison Bottle,2,5000,,100,,,,,0xFFFFFFFF,7,2,,,,,,{ if(Class==Job_Assassin_Cross) { sc_start SC_DPOISON,60000,0; sc_start SC_ATTHASTE_INFINITY,60000,0; } else percentheal -100,-100; },{},{}

    changes to:

    {
    Id: 678
    AegisName: "Poison_Bottle"
    Name: "Poison Bottle"
    Type: 2
    Buy: 5000
    Weight: 100
    Script: <"
        if(Class==Job_Assassin_Cross) {
            sc_start SC_DPOISON,60000,0;
            sc_start SC_ATTHASTE_INFINITY,60000,0;
        }
        else percentheal -100,-100;
    ">
},

    Better, isn't it!? Let's now try a Jellopy:

    909,Jellopy,Jellopy,3,6,,10,,,,,,,,,,,,,,{},{},{}

    Count the commas! Did I miss any? An extra comma you say? No, I don't want to count them, thanks...

    {
    Id: 909
    AegisName: "Jellopy"
    Name: "Jellopy"
    Buy: 6
    Weight: 10
},

    Not even a comma!
    Now, help me read what this item does?

    13307,Krieger_Huuma_Shuriken1,Glorious Shuriken,4,20,,0,55,,1,0,0x02000000,7,2,34,4,80,1,22,{ bonus2 bAddRace,RC_DemiHuman,95; bonus2 bIgnoreDefRate,RC_DemiHuman,20; bonus bMatkRate,15; autobonus "{ bonus2 bSkillAtk,NJ_HUUMA,100; bonus2 bSkillAtk,NJ_ISSEN,100; }",50,10000; bonus bUnbreakableWeapon,0; if(getrefine()>5) { bonus2 bAddRace,RC_DemiHuman,(getrefine()-3)*(getrefine()-3); bonus2 bIgnoreDefRate,RC_DemiHuman,5; } if(getrefine()>8) { bonus5 bAutoSpellOnSkill,NJ_ISSEN,AL_HEAL,10,1000,1; bonus4 bAutoSpellOnSkill,NJ_HUUMA,NPC_CRITICALWOUND,2,200; } },{},{}
    {
    Id: 13307
    AegisName: "Krieger_Huuma_Shuriken1"
    Name: "Glorious Shuriken"
    Type: 4
    Buy: 20
    Atk: 55
    Range: 1
    Job: 0x02000000
    Loc: 34
    WeaponLv: 4
    EquipLv: 80
    View: 22
    Script: <"
        bonus2 bAddRace,RC_DemiHuman,95;
        bonus2 bIgnoreDefRate,RC_DemiHuman,20;
        bonus bMatkRate,15;
        autobonus "{ bonus2 bSkillAtk,NJ_HUUMA,100; bonus2 bSkillAtk,NJ_ISSEN,100; }",50,10000;
        bonus bUnbreakableWeapon,0;
        if(getrefine()>5) {
            bonus2 bAddRace,RC_DemiHuman,(getrefine()-3)*(getrefine()-3);
            bonus2 bIgnoreDefRate,RC_DemiHuman,5;
        }
        if(getrefine()>8) {
            bonus5 bAutoSpellOnSkill,NJ_ISSEN,AL_HEAL,10,1000,1;
            bonus4 bAutoSpellOnSkill,NJ_HUUMA,NPC_CRITICALWOUND,2,200;
        }
    ">
},

    Which one did you find easier to read? Old or new format?
     
    Want to see an example that specifies both min and max levels? Here you go:

    {
    Id: 2819
    AegisName: "Swordman_Manual"
    Name: "Swordman Manual"
    Type: 5
    Buy: 0
    Weight: 100
    Job: 0x00000001
    Upper: 47
    Loc: 136
    EquipLv: [1, 12]
    Refine: false
    Script: <"
        bonus bMaxSP,100;
        skill SM_BASH,1;
        skill SM_PROVOKE,1;
        skill SM_MAGNUM,1;
    ">
},

    Okay, one last example. Let's try to make an item_db2 entry for a Pupa Card that gives a bonus of 1000 HP rather than just 700.

    {
    Id: 4003
    Inherit: true
    Script: <" bonus bMaxHP,1000; ">
},

    Done. No need to rewrite the name, location, prices... those are already in the item_db!
     
    But... I have several custom items, do I have to manually convert all of them...?
     
    Of course you do! No, I'm kidding, don't hate me :(
     
    It's true that you need to convert your item database to the new format, but we can do it for you!

    • Go to http://haru.ws/hercules/itemdbconverter/ and paste (or upload) your item_db2.txt (or even your item_db.txt if you have custom entries there), press the Convert button, wait a few seconds and you're done! It's already converted for you. Easy, isn't it?
    • Don't trust us? No, no, we don't need your custom items, you can sleep safe... But if you still don't want to paste anything on a website... well, we have provided the source code of the converter script! It's in the 'tools' folder of the Hercules repository. All you need is a Perl interpreter (and if you're running Linux or Mac OS, on either your server or your own computer, it's almost certain that you already have that). All you have to do is run it (example: perl tools/itemdbconverter.pl < db/item_db2.txt > db/item_db2.conf), and your item database will be converted in a split second!

    What if I was using SQL item databases?
     
    Well... Then you won't have to worry about the txt databases, unless you were creating the items in txt, then converting them to sql (if you're not doing this, maybe you should consider it, you know? It's easier to create and manage them in txt even if you use the sql version!)
     
    If you were converting your custom item databases from the txt version to sql through the db2sql plugin, there are good news for you! We just updated the plugin, and now it's able to create a .sql script rather than just inserting the entries into your database. This means that you can create the SQL entries in your own computer, without even needing a database (just a copy of Hercules), without risking to slow down your server, and then upload them whenever you want.
     
    And if you're relying on the .sql scripts we provide... well, there are still good news for you! Outdated item_db.sql files are a thing of the past now: those scripts will get updated automatically after every commit by our HerculesWS API bot, exactly like the HPM Hooks!
     
    As a final note, now the pre-re and re databases share the same structure in SQL as well! no more column mismatch if you're trying to import into your Renewal server an item that was meant for pre-renewal.
     
    If you have data already imported to your item_db2 SQL table, it'll be updated to the new table format through the regular upgrade sql mechanism (just run the provided script in the sql-files/upgrades folder). Please note that the script requires MySQL 5.0 or newer -- if you're still on MySQL 4.0, please open it in a text editor, read the comments and run the provided queries manually.
     
    I have this event item entry that came with an old script I downloaded...
     
    No worries, you can get it converted. Use the same script (or the provided web page) you'd use to convert an entire item database, it'll work just fine even for an item or two!
     
    Special thanks

    • To Ind, for bringing up the idea and pushing it forward until it was implemented. And for his awesome work on the HerculesWS API and database converter plugin.
    • To Yommy, for the original idea, and for some invaluable help finalizing the actual database format.

    Links~u!

    TecnoCronus, Mystery, Mhalicot and 10 others reacted to this

  16. Posted

    yes, but this packet keys do not work, i can still spam with WPE & other toolz.

    This is because:

    Yes, the 2012-04 clients (and possibly a few others) have their keys chosen in such a way that makes the client effectively always use the same encryption key for each sent packet (thus allowing you to replay a packet as much as you want in WPE.)

     

    You can either use a newer client, or edit the keys yourself in both client (you'll have to hex-edit it) and server (edit the appropriate packetKeys line for your PACKETVER in src/map/packets.h and recompile.)

     

  19. Posted

    To finish, it's not because you don't know or heard about exploits that they don't exist.

    Lucky admins, RO hackers seems to have a pretty bad level and just know about SQLI, LFI (and DDOS of course) :)

    How would a full rewrite help preventing these attacks? I feel that ditching an old (but maintained) and well tested codebase to rewrite it from scratch, would have the opposite effect and risk creating more security vulnerabilities, other than breaking compatibility with any old addons and themes. As someone once said, if it ain't broke, don't fix it.
    GauxeVaugh reacted to this

  20. Posted

    Yes, the 2012-04 clients (and possibly a few others) have their keys chosen in such a way that makes the client effectively always use the same encryption key for each sent packet (thus allowing you to replay a packet as much as you want in WPE.)

     

    You can either use a newer client, or edit the keys yourself in both client (you'll have to hex-edit it) and server (edit the appropriate packetKeys line for your PACKETVER in src/map/packets.h and recompile.)

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.

  I accept