Upcoming Upgrades

Discussion in 'Admin News' started by dimopoulos, Mar 14, 2015.

  1. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    Ladies,

    We have been all experiencing issues with the forum mainly with the search engine. I understand you are frustrated and trust me so am I, because the performance of the forum has not been stellar the last couple of months.

    For the last few months we have been slowly upgrading our hardware and software on the servers (that run this forum) and in the next couple of weeks we will upgrade the forum software.

    There are plenty of reasons for the upgrade and changing the forum software. Some are:
    - Better feature set
    - Up to date software with more security features
    - More responsive design - no need for a mobile app (our vendor has not updated the mobile app for over a year and they do not intend on updating it)
    - Better search
    - Better photo gallery
    and more.

    This weekend I am running a test upgrade to identify potential issues. This will not interfere with the forum operations for the most part.

    Due to the size of our forum and the database, there is a lot of data that needs to be transferred back and forth in different databases to make this test upgrade happen. I will try to ensure that there is no downtime more than necessary.

    Thank you all for your understanding. I will keep you updated on the progress
     
  2. JustSitNBePretty

    JustSitNBePretty Well-Known Member

    Blog Posts:
    0
    Joined:
    Oct 24, 2014
    Messages:
    1,717
    Likes Received:
    9,348
    Trophy Points:
    113
    We've been hearing words like "in the next couple months" and "in the next couple weeks" for the longest as it pertains to an upgrade. Can we get something a bit more specific? At this point, we know that "the next couple months" for you could also mean December 2015.
     
  3. Rsgal

    Rsgal Well-Known Member

    Blog Posts:
    0
    Joined:
    Feb 16, 2009
    Messages:
    577
    Likes Received:
    1,944
    Trophy Points:
    93
    Thank you for finally informing us of what is going on, frustrating as it is!
     
  4. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    JustSitNBePretty I apologize for being extra cautious and indeed vague in a timeframe. My main concerns are a) ensure that we have no loss of data and b) ensure that we don't have any loss of functionality.

    Our database is over 28GB holding more than 11M posts. I have to perform a couple of test upgrades first before the real thing. There are a lot of modifications in our forum that offer the existing functionality that have to be adjusted or created from scratch before anything happens.

    At the moment we cannot run the test upgrade on the same environment because we will run out of space on our database environment (hence the hardware upgrade that will happen at the same time) For that we had to create a new environment that will host the upgraded database, and if everything goes well, the forum itself for a couple of weeks until our primary servers are upgraded both in hardware as well as software.

    I cannot simply shut the forum down to do either the test or the live upgrade so I have to do the whole task in steps, which at the moment I don't know how long it will take.

    The easiest example I can give you is taking a backup of the database takes over 3 hours. That database has to be restored somewhere else, the upgrade to run (my test upgrade initially took over a day) etc.

    If I have a more definitive timeline by all means I will share it but for the time being that is the best I can do.
     
    Last edited: Mar 17, 2015
  5. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    Minor update on the progress

    During the weekend our database experienced a crash (it happens) and we had to repair it. No data was lost but we had to shut the forum down for an hour to do the necessary repairs.

    Following that I managed to get a copy of our database and transfer it to a new environment for testing purposes.

    - The installation of the new forum software (in that test environment) was very easy and took very little time to be completed.
    - The transfer of the database backup to the new location over took just short of 60 minutes
    - It took three hours to restore the db in the new environment
    - I am still running the importer (transferring data from our current forum to the new forum software). I am waiting for the posts to be transferred (at 10% at the moment).

    Some stats:
    Import users: 3 hours
    Import private messages: 3.5 hours
    Import forums, profile comments etc. 4 hours
    Import posts.... still waiting. At 10% running 2 hours now.

    When we do the upgrade it will most likely be a whole day event. I will try and shorten the down time as much as possible but at least the above should give you some idea of what to expect.
     
  6. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    Importing posts still running.

    At 71% at the moment 20.5 hours elapsed
     
  7. january noir

    january noir Sunny On a Cloudy Day

    Blog Posts:
    0
    Joined:
    Oct 9, 2006
    Messages:
    24,430
    Likes Received:
    52,171
    Trophy Points:
    113
    Location:
    Along the 287th Parallel
    Thanks for the status report.
     
  8. sj10460

    sj10460 Don't Come for me unless I send for you!

    Blog Posts:
    0
    Joined:
    Dec 25, 2006
    Messages:
    17,775
    Likes Received:
    80,439
    Trophy Points:
    113
    Location:
    I'm from The Bronx!!
    dimopoulos are the crashes a result of high traffic? I've notice I can't log-on certain days when there's a popular show televised. This past Wednesday (yesterday) I was unable to log on for over an hour.

    Is it possible to just shut the site down for complete day or during a timeframe ie Saturday & Sunday 12am-12pm and work on the required maintenance rather then shutting down an hour here and there? I think the former would be most effective.
     
  9. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    The posts have been imported in the test installation. That task alone took 38 hours and change :(

    Finishing the import today.


    sj10460 No the crash on the database was something that can happen any day. We are prepared for that with replicated databases and the repair function works just fine. It is the size of our database that makes things take longer than normal.

    Irrespective of what is going on elsewhere (shows, grammys, oscars, speeches, other events) that stimulate more traffic for this site, we have not seen spikes on our server cluster that we cannot handle. The only thing that we need to address is the search (and we are doing this with this upcoming upgrade) which cannot keep up any more with the site's growth.

    One other thing we will do (because we have observed this) is when the site goes down for maintenance or otherwise, visitors will be redirected to a brand new domain. This way your browser will not have the cached maintenance page and showing you erroneous information i.e. the site is up and you still see the maintenance page.



     
  10. Duchess007

    Duchess007 Well-Known Member

    Blog Posts:
    0
    Joined:
    Nov 11, 2012
    Messages:
    1,499
    Likes Received:
    3,780
    Trophy Points:
    113
    Location:
    Madison, WI
    dimopoulos Is photo album functionality something that is down for the upgrade, or is this gone forever?
     
  11. nathansgirl1908

    nathansgirl1908 Well-Known Member

    Blog Posts:
    0
    Joined:
    May 29, 2009
    Messages:
    21,752
    Likes Received:
    95,331
    Trophy Points:
    113
    Location:
    Timmy
    I agree with the previous suggestion by SJ. Why not just shut the forum down and do what needs to be done? When I posted on TPf, Vlad would just shut down the forum and do the updates. Sometimes it took a few days but we survived. And we can survive on here too. It would be worth it in order to have the forum running smoothly.
     
  12. sheanu

    sheanu Well-Known Member

    Blog Posts:
    0
    Joined:
    Mar 21, 2011
    Messages:
    1,869
    Albums:
    1
    Likes Received:
    6,360
    Trophy Points:
    113
    Yup I completely agree. Just rip off the bandaid lol
     
  13. HairNinja

    HairNinja Well-Known Member

    Blog Posts:
    0
    Joined:
    Apr 16, 2012
    Messages:
    935
    Likes Received:
    2,904
    Trophy Points:
    93
    He already stated the answer to this question.

    dimopoulos
    "I cannot simply shut the forum down to do either the test or the live upgrade so I have to do the whole task in steps, which at the moment I don't know how long it will take."
     
  14. nathansgirl1908

    nathansgirl1908 Well-Known Member

    Blog Posts:
    0
    Joined:
    May 29, 2009
    Messages:
    21,752
    Likes Received:
    95,331
    Trophy Points:
    113
    Location:
    Timmy
    Yeah but is that because he knows people would be upset or because he CAN'T? Like I said, Vlad did it for TPF.
     
  15. brooklyngal73

    brooklyngal73 SteelyDan/DonaldFagenFan!

    Blog Posts:
    0
    Joined:
    Oct 7, 2005
    Messages:
    29,202
    Likes Received:
    37,643
    Trophy Points:
    113
    Location:
    In ET/OT land for kicks....
    Exactly! That point wasn't clarified.

     
  16. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    To clarify:

    For the upcoming upgrade there are some tasks that need to be completed while the forum is running as usual that do interfere with the operation of the forum. Other tasks are independent of that.

    For instance, to take a database backup so that I can run the test upgrade, that interferes with the normal forum operation since it increases the load on the servers and makes the forum slower. That step has already been completed.

    To run the import on a different environment is pretty straight forward and does not interfere with the normal operation of the forum. That I have been doing for the last few weeks now.

    I have run the import a number of times now, learning what obstacles I will face when we do the real thing. For instance one thing that I found out is that the @ mentions requires some special handling. Also I am not satisfied with the import/migration of the "thank you". The totals are there but you cannot see who actually thanked your post. That is what I am working on now.

    When I am 100% sure then we will upgrade. At that time the main forum will close and will remain closed for the duration of the upgrade, which as it seems now it will be a couple of days.

    There is no point in me switching the forum off for a day or two just to run a test upgrade. That is why I am splitting the task in stages to get the know-how so to speak and to make sure that it is business as usual for the main forum.

    I will however have to speed up the process significantly since as it seems our search engine has been acting up a lot lately which removes from the user experience.
     
  17. sugarbaybie

    sugarbaybie Well-Known Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2013
    Messages:
    5,746
    Likes Received:
    11,561
    Trophy Points:
    113
    dimopoulos why do I keep getting logged off every 3-5 seconds. Yes. Literally.


     
  18. JustSitNBePretty

    JustSitNBePretty Well-Known Member

    Blog Posts:
    0
    Joined:
    Oct 24, 2014
    Messages:
    1,717
    Likes Received:
    9,348
    Trophy Points:
    113
    No one really comes into this part of the forum. You need to make a banner saying the forum will be down for a couple days in the coming weeks due to upgrading, that way people won't be caught off guard and know what's going on. Poor communication is partly why so many of us are irritated and annoyed. I also have a feeling "a couple days" really means two weeks.
     
  19. brooklyngal73

    brooklyngal73 SteelyDan/DonaldFagenFan!

    Blog Posts:
    0
    Joined:
    Oct 7, 2005
    Messages:
    29,202
    Likes Received:
    37,643
    Trophy Points:
    113
    Location:
    In ET/OT land for kicks....
    I've been repeatedly saying this. Things pertaining to search, etc. should be an announcement that's in all forums (via sticky/banner). This section is virtually tumbleweed central vs. the entertainment/OT sections.

    :nono:

     
  20. Kiowa

    Kiowa Well-Known Member

    Blog Posts:
    1
    Joined:
    Dec 19, 2005
    Messages:
    18,554
    Likes Received:
    113,443
    Trophy Points:
    113
    Location:
    'Stolen From Africa,and living in Babylon"
    Thanks Nikos..I just completed an upgrade using an import/export method for one of my clients...these can be a PITA, our first run went south, I was skeptical of our backup, and a lot of long days...:nono: needless to say, it ain't easy running doing a forum, but thanks for sharing those updates..
     
  21. dimopoulos

    dimopoulos Crazy Greek Staff Member

    Blog Posts:
    0
    Joined:
    Dec 16, 2004
    Messages:
    3,442
    Media:
    2
    Albums:
    1
    Likes Received:
    1,546
    Trophy Points:
    113
    Location:
    Charles Town, WV
    Duchess007 No the photos will all be imported. Each will have their own album and you will be able to create as many albums as you want

    sugarbaybie Not sure why this would be the case. It could be your computer, your network or our server. Please bare with us until next week where we will be on the new platform and see if the issue persists. If not all good, if it does, please open a support ticket and we can troubleshoot it together.

    JustSitNBePretty This is the announcement area of upcoming upgrades or any admin news. It is unrealistic to expect a private message every time something happens or an announcement for something that might/will happen in the future. If members do not read this section there is nothing I can do.

    Every forum I know has a similar section and members do look at it from time to time. Adding an announcement (which we already put up there) is only for things that impact the forum i.e. the upgrade, not for things that are planned when we don't have a time frame established.

    As you see now that we do have a target date, the announcement is up. Good call though, we think alike it seems :) As for the couple of days, see my other post with how much time it took to run the test upgrade. It was just shy of 48 hours so yes it is a couple of days.
     
  22. JustSitNBePretty

    JustSitNBePretty Well-Known Member

    Blog Posts:
    0
    Joined:
    Oct 24, 2014
    Messages:
    1,717
    Likes Received:
    9,348
    Trophy Points:
    113
    Woah, when did I say anything about sending people a private message? I said you should make an announcement BANNER that appears at the top of the page saying that the forum will be down for 2 days for an upgrade.

    What do you mean, "if members don't read this section there's nothing you can do about it?" Are you serious? This isn't a popular part of the forum and no one really comes in this section because issues are resolved at snail's pace. This is clearly a popular board. I think better efforts can be made to inform paying members instead of making this out to be our fault.
     
  23. PatDM'T

    PatDM'T Well-Known Member

    Blog Posts:
    0
    Joined:
    Aug 10, 2009
    Messages:
    352
    Likes Received:
    2,158
    Trophy Points:
    93
    dimopoulos, can you maybe put the "forum down" announcement
    the same place where the notice for search function down is now?
    I would of been wondering why you removed the search feature
    Were it not for that conspicuous sign
    That very obvious announcement done tell me all I needed to know

    I think a sign about the forum going down in that spot would be helpful
     
  24. SuchaLady

    SuchaLady Well-Known Member

    Blog Posts:
    0
    Joined:
    May 23, 2010
    Messages:
    15,633
    Likes Received:
    91,079
    Trophy Points:
    113
    Location:
    GA❤️
  25. lonelysky

    lonelysky Baby Chick Gate 2011

    Blog Posts:
    0
    Joined:
    Nov 7, 2005
    Messages:
    7,326
    Likes Received:
    20,532
    Trophy Points:
    113
    Did I understand that correctly? Will there no longer be an app?
     
  26. PatDM'T

    PatDM'T Well-Known Member

    Blog Posts:
    0
    Joined:
    Aug 10, 2009
    Messages:
    352
    Likes Received:
    2,158
    Trophy Points:
    93
    dimopoulos, I love they way you pin the upgrade announcement link at the top of every forum.
    Many thanks!
     
  27. january noir

    january noir Sunny On a Cloudy Day

    Blog Posts:
    0
    Joined:
    Oct 9, 2006
    Messages:
    24,430
    Likes Received:
    52,171
    Trophy Points:
    113
    Location:
    Along the 287th Parallel
    It used to be done all the time. I don't know why it stopped.
     

Share This Page